Bug 87695 – qmail tarpitting before relay-ctrl

• Product: Gentoo Linux
• Status: VERIFIED INVALID

[Description Jason Carlson 2005-04-02 11:06:57 UTC]

I can't determine if tarpitting is an actual patch applied (as it is not listed but a few patches appear to be collections of patches) but I am getting a behaviour that is as if tarpitting was patched improperly.

I have remote users and depend on relay-ctrl to authorizing them access after they check email.  Many users are dialup users and thus right their emails offline and then connect to send.  If their email program attempts to send the message before they checked (thus not approved by relay-ctrl) they never connect to the mailserver.  My firewall shows approved and active connections from these users to the mailserver but the mailserver is silently ignoring them (like tarpitting would do if it suspected these people as trying to illegally relay).

Even if the users cancel the request to send the message, check their emails, are approved by relay-ctrl for their next send attempt they are still being silently dropped for the few minutes.

This only happens to them when their email program tries and send an email before it checks them (which is the behaviour of most email client programs).

It is only my dialup clients who write emails offline that are running into this problem.

Reproducible: Always
Steps to Reproduce:
1. write an email offline
2. connect to internet (dialup clients generally) and click send/recieve (Outlook for example will send messages in outbox before attempting to read emails)

3.

Actual Results:  
Users are tarpitted (or at least that behaviour where attempts to connect are
silently dropped as if mailserver is down).  They remain locked out for some
time even if after the sending fails, they check the email and are listed as
approved in the relay-ctrl allow files.

Expected Results:  
Legit users should not be tarpitted.  I imagine because a connection was made to
the mailserver that is dropped even once they are approved the tarpitting stays
in effect (possibly as it reuses the same connection??).

tarpitting should be an option that can be disabled so businesses such as ours
with lots of dialup clients can actually use this package.

Side note:

I have a client base that changes often and thus have hundreds of email
addresses for my domains that do not exist anymore yet get dozens of spams on
these.  In fact I am getting approxiamately 60 emails a minute that are for
accounts that do not exist anymore.

Before switching our business from RedHat Enterprise to Gentoo we had qmail with
the smtpd-chkusr patch which prevented our servers from ever getting any of
these emails on our system (stopped it before bodies of messages were even sent
to us).  Adding this patch saved me between 85%-90% of bandwidth costs
(financial and load).

This patch seems to have been never put in and reading bug number 40486 it looks
like developers are opting for SPP which needs additional configurations yet I
have not found any clear instructions on how to do this.

Can someone point me in the right direction to get this feature enabled (or
consider applying the new smptd-chkuser v2 patch (which looks very flexible like
spp but without any special configuration it always rejects all emails to people
who do not exist for your registered domains).

I've listed this as major as it's forcing me to move my clients back onto RedHat
boxes as we were having too many clients thinking our servers were down and
explaining tarpitting is beyond most clients comprehension.

[Comment 1 Robin Johnson 2005-04-04 21:02:29 UTC]

this may come as a surprise, but we don't have any tarpit support in qmail up to and including -r15. (it's a possibility for -r16, but it is disabled by default there).

I can't reproduce the problem as you describe it here (but I don't have a dialup account anywhere, so I was just using a machine at work).

Most mail clients (at least when I've used them) do support checking before sending, and SMTP AUTH is available as well (a much better solution than POP/IMAP-before-SMTP anyway).

Could you please provide more details, incl. your 'emerge -v info' output, and versions of all relevant packages along with any modified configurations (modified from the Gentoo stock config).

[Comment 2 Jason Carlson 2005-04-07 21:47:22 UTC]

I couldn't find a separate tarpit patch that was applied so I wasn't surprised yet the behaviour is as if one is installed so I wondered if one of the those patches that appear to apply a few combined fixes for older fixes contained one.  Now sure why this is happening, but it still is.  Perhaps the best solution is to keep digging for some accurate documentation on how to implement the SMTP AUTH patch that is already built in (just not activated - so far docs I've found have not helped me get that going).

Here is what I have installed now.

Version installed:
qmail-1.03-r15
vpopmail-5.4.6-r1
courier-imap-4.0.1-r1
qmail-autoresponder-0.96.1-r1
relay-ctrl-3.1.1-r2
ezmlm-idx-mysql-0.40-r2

emerge -v info:
Gentoo Base System version 1.4.16
Portage 2.0.51.19 (default-linux/x86/2004.3, gcc-3.4.3, glibc-2.3.4.20040808-r1, 2.6.10-gentoo-r6 i686)
=================================================================
System uname: 2.6.10-gentoo-r6 i686 Intel(R) Pentium(R) 4 CPU 1.80GHz
Python:              dev-lang/python-2.3.4-r1 [2.3.4 (#1, Feb  8 2005, 16:38:26)]
distcc 2.16 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632) [enabled]
ccache version 2.3 [enabled]
dev-lang/python:     2.3.4-r1
sys-devel/autoconf:  2.59-r6, 2.13
sys-devel/automake:  1.7.9-r1, 1.8.5-r3, 1.5, 1.4_p6, 1.6.3, 1.9.4
sys-devel/binutils:  2.15.92.0.2-r1
sys-devel/libtool:   1.5.10-r4
virtual/os-headers:  2.6.8.1-r2
ACCEPT_KEYWORDS="x86"
ACCEPT_LICENSE=""
ARCH="x86"
AUTOCLEAN="yes"
BASH_ENV="/etc/spork/is/not/valid/profile.env"
CCACHE_SIZE="2G"
CFLAGS="-march=pentium4 -O3 -pipe -fomit-frame-pointer"
CHOST="i686-pc-linux-gnu"
CLEAN_DELAY="5"
CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3/share/config /usr/share/config /var/qmail/alias /var/qmail/control /var/vpopmail/domains /var/vpopmail/etc"
CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d"
CVS_RSH="ssh"
CXXFLAGS="-march=pentium4 -O3 -pipe -fomit-frame-pointer"
DCCC_PATH="/usr/lib/distcc/bin"
DISTCC_LOG=""
DISTCC_VERBOSE="0"
DISTDIR="/usr/portage/distfiles"
EDITOR="/usr/bin/vim"
FEATURES="autoaddcvs autoconfig ccache distcc distlocks sandbox sfperms"
FETCHCOMMAND="/usr/bin/wget -t 5 --passive-ftp -P ${DISTDIR} ${URI}"
GCC_SPECS=""
GENTOO_MIRRORS="http://mirror.datapipe.net/gentoo ftp://gentoo.agsn.ca/ http://gentoo.seren.com/gentoo http://gentoo.chem.wisc.edu/gentoo/"
GRP_STAGE23_USE="ipv6 pam tcpd readline nls ssl gpm perl python berkdb ncurses"
HOME="/root"
HOSTNAME="mail"
INFODIR="/usr/share/info"
INFOPATH="/usr/share/info:/usr/share/gcc-data/i686-pc-linux-gnu/3.4.3/info"
INPUTRC="/etc/inputrc"
LESS="-R"
LESSOPEN="|lesspipe.sh %s"
LOGNAME="root"
LS_COLORS="no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=01;32:*.cmd=01;32:*.exe=01;32:*.com=01;32:*.btm=01;32:*.bat=01;32:*.sh=01;32:*.csh=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.mng=01;35:*.xcf=01;35:*.pcx=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.avi=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.mov=01;35:*.qt=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.fli=01;35:*.gl=01;35:*.dl=01;35:*.pdf=00;32:*.ps=00;32:*.txt=00;32:*.patch=00;32:*.diff=00;32:*.log=00;32:*.tex=00;32:*.doc=00;32:*.mp3=00;36:*.wav=00;36:*.mid=00;36:*.midi=00;36:*.au=00;36:*.ogg=00;36:*.flac=00;36:*.aac=00;36:"MAIL="/var/mail/root"MAKEOPTS="-j3"MANPATH="/usr/share/man:/usr/local/share/man:/usr/share/gcc-data/i686-pc-linux-gnu/3.4.3/man"NOCOLOR="false"PAGER="/usr/bin/less"PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/opt/bin:/usr/i686-pc-linux-gnu/gcc-bin/3.4.3:/var/qmail/bin:/var/vpopmail/bin"PKGDIR="/usr/portage/packages"PORTAGE_ARCHLIST="alpha amd64 arm hppa ia64 m68k mips ppc ppc64 ppc-macos ppc-od s390 sh sparc x86 x86-fbsd x86-obsd x86-od"PORTAGE_BINHOST_CHUNKSIZE="3000"PORTAGE_CALLER="emerge"PORTAGE_GID="250"PORTAGE_MASTER_PID="20049"PORTAGE_TMPDIR="/var/tmp"PORTDIR="/usr/portage"PRELINK_PATH=""PRELINK_PATH_MASK=""PS1="\[\033[01;32m\]\u@\h \[\033[01;34m\]$(/bin/pwd)\$ \[\033[00m\]"PWD="/root"QMAIL_CONTROLDIR="/var/qmail/control"RESUMECOMMAND="/usr/bin/wget -c -t 5 --passive-ftp -P ${DISTDIR} ${URI}"RPMDIR="/usr/portage/rpm"RSYNC_RETRIES="3"RSYNC_TIMEOUT="180"SHELL="/bin/bash"SHLVL="1"SSH_CLIENT="hidden ip 41180 22"SSH_CONNECTION="hidden ip 41180 hidden ip"SSH_TTY="/dev/pts/0"SYNC="rsync://petra2/portage"TERM="xterm"USE="x86 alsa apm arts authdaemond avi berkdb bitmap-fonts crypt cups emboss encode font-server foomaticdb fortran gdbm gif gtk2 imlib jpeg libg++ libwww mad maildir mikmod motif mp3 mpeg mysql ncurses nls oggvorbis opengl oss pam pdflib perl png python quicktime readline sdl spell ssl svga tcpd tiff truetype truetype-fonts type1-fonts valias vhosts xml2 xmms xv zlib"
USER="root"
USERLAND="GNU"
USE_EXPAND="VIDEO_CARDS INPUT_DEVICES LINGUAS"
XARGS="xargs -r"
_="/usr/bin/emerge"



in /var/qmail/control/conf-common the only line changed was the TCPSERVER_OPTS line which is now set to: 
TCPSERVER_OPTS="-H -R -l 0"
(believe if was -p -v before)


in /var/qmail/control/conf-smtpd file now reads as: (basically just uncommented sections that applied and made it to work with vpopmail).
# Configuration file for qmail-smtpd
# $Header: /var/cvsroot/gentoo-x86/mail-mta/qmail/files/conf-smtpd,v 1.4 2005/01/28 08:15:18 hansmi Exp $

# Stuff to run before tcpserver
#QMAIL_TCPSERVER_PRE=""
# Stuff to run qmail-smtpd
#QMAIL_SMTP_PRE=""
# Stuff to after qmail-smtpd
#QMAIL_SMTP_POST=""

# this turns off the IDENT grab attempt on connecting
TCPSERVER_OPTS="${TCPSERVER_OPTS} -R"

# fixcrio inserts missing CRs at the ends of lines. See:
# http://cr.yp.to/ucspi-tcp/fixcrio.html
# http://cr.yp.to/docs/smtplf.html
# DO NOT enable this, when you are using SSL/TLS (USE=ssl)!
#QMAIL_SMTP_PRE="${QMAIL_SMTP_PRE} fixcrio"

# You might want to use rblsmtpd with this, but you need to fill in a RBL server here first
# see http://cr.yp.to/ucspi-tcp/rblsmtpd.html for more details
QMAIL_SMTP_PRE="${QMAIL_SMTP_PRE} rblsmtpd -r sbl.spamhaus.org"

# If you are interested in providing POP or IMAP before SMTP type relaying,
# emerge relay-ctrl, then uncomment the next 2 lines
QMAIL_TCPSERVER_PRE="${QMAIL_TCPSERVER_PRE} envdir /etc/relay-ctrl relay-ctrl-chdir"
QMAIL_SMTP_PRE="${QMAIL_SMTP_PRE} relay-ctrl-check"
# In /etc/courier-imap/authdaemonrc add the next line to the end:
#authmodulelist="${authmodulelist} relay-ctrl-allow"
# Then in /etc/courier-imap/{imapd,imapd-ssl,pop3d,pop3d-ssl}
# Add this at the end
#PRERUN="${PRERUN} envdir /etc/relay-ctrl relay-ctrl-chdir"

# This next block is for SMTP-AUTH
# This provides the LOGIN, PLAIN and CRAM-MD5 types
# the 'cmd5checkpw' used in $QMAIL_SMTP_AUTHCHECKPASSWORD supports CRAM-MD5
# and reads it's data from /etc/poppasswd
# see the manpage for cmd5checkpw for details on the passwords
# uncomment the next four lines to enable SMTP-AUTH
QMAIL_SMTP_AUTHHOST=$(<${QMAIL_CONTROLDIR}/me)
[ -z "${QMAIL_SMTP_POST}" ] && QMAIL_SMTP_POST=/bin/true
#QMAIL_SMTP_CHECKPASSWORD="/bin/cmd5checkpw"
QMAIL_SMTP_CHECKPASSWORD="/var/vpopmail/bin/vchkpw"
QMAIL_SMTP_POST="${QMAIL_SMTP_AUTHHOST} ${QMAIL_SMTP_CHECKPASSWORD} ${QMAIL_SMTP_POST}"

Do you need the /etc/courier and /etc/courier-imap files too?  POP3 works fine and when a user checks there email, I see their IP in the relay-ctrl/allow directory.  If they tried to send an email before checking, they are blocked as they should be and once they check their IP is listed in the relay-ctrl/allow directory but they still cannot send and remain locked out for at least 10 minutes.  Firewall shows they have an approved connection to the mailserver.  Mailserver doesn't respond (times out - exactly as if there was a tarpit patch applied).  I don't know exactly how long the block happens (users have only seemed to try 10 minutes before giving up), but it does work again when they try about half an hour from there first attempt.

[Comment 3 Robin Johnson 2005-04-07 22:13:23 UTC]

"Firewall shows they have an approved connection to the mailserver.  Mailserver doesn't respond (times out - exactly as if there was a tarpit patch applied)."
That's very strange. Are you 100% sure that the mail server doesn't respond at all? Use tcpdump with a test user and replicate the problem yourself, and attach the dumpfile if there is actually traffic flowing.

Also turn off the RBL stuff in your config file for testing this, and see if that makes a difference.

[Comment 4 nuitari 2005-04-11 02:48:45 UTC]

Similar behavior does happen if there is a very slow to respond RBL server. If there is one that you defined that regularly takes a lot of time to answer it might be better to drop it or try to download their zone file and run your own rbldns server, if they allow it.

[Comment 5 Jason Carlson 2005-04-12 13:00:20 UTC]

Sorry for the delay.  Life got busy for a bit.  

I had already tried disabling RBL all together and this problem still existed.  I haven't had a chance to sniff traffic to see if the server is really not responding.

This became a less priority as I suggested to my dialup clients (again) that if they want this problem to go away to use their dialup providers smtp server.  So I am finally not getting complaints about this.

I will see if I can simulate this behaviour using manual connections via telnet and if so sniff the traffic to/from the mailserver to see if I can give you more detail.

As a side note.  I noticed that when relay-ctrl logs an IP address to allow it to relay that it does not also turn off RBL checks for that connection so those clients have a slight delay each time the RBL check occurs.  The IP roaming feature of vpopmail that I used to use (before switching to gentoo) stopped RBL checks from approved IP addresses.  This isn't really important as I am going to be just be leaving RBL checks completely off now as the very conservative list I use is only blocking a tiny percentage of spams and I don't think it's worth the extra overhead.  More agressive RBL providers tend to block too many legit mails to even consider them.

[Comment 6 Michael Hanselmann (hansmi) (RETIRED) 2005-08-07 03:51:28 UTC]

A few things, patches included, have changed in the qmail-1.03-r16 ebuild. Can
you please try it again with it and report back?

[Comment 7 Jason Carlson 2005-08-08 09:26:40 UTC]

Sorry because I got my clients to use their own internet providers smtp servers
I didn't have the critical fix needed so this got put on the bottom of the todo
list.  Getting the most recent update I started probing into this again.

Intially both ADSL clients and dialup clients complained of this problem but now
it appears ADSL clients can switch to my smtp server and get a connection. 
Dialup clients get unusal tarpitting type connections.  This leads me to believe
that it is NOT gentoo's qmail package giving me the problem but actually the
provider.  The dialup provider in our area insists they are not blocking
anything and that the problem must be with our mailserver.  However I got an old
friend across the country with a different dialup provider to try it out (I only
have one in my area) and they did not have any problems connecting.

So it has to be our local dialup provider (telus.net).  It's not a complete
block as if I use telnet to test a smtp connection I get the initial connection
success message but then everything after that is timing out (silently dropped
is my guess).  Thus what gives a tarpit type behaviour, not a complete firewall
block.  If I try connecting again while on the same dialup connection I cannot
access my own or any other smtp server even for the initial connection so it is
definately the dialup provider who has some sort of blocking system in place for
smtp connections other than their own.

My guess is the ADSL provider (same company as the dialup) was blocking ADSL
connections to outside smtp servers but must have since removed that as those
clients can now connect just fine.  The problem only remains with dialup clients
from the same provider.  The simple fix was to have the dialup clients use their
dialup providers smtp server for outgoing mail.

The fact I could always see initial connections to the mailserver (thus not a
full block) and the fact the internet providers insited they were not blocking
it in anyway that I assumed it was some tarpitting problem with gentoo's qmail
installation.  So it seems to come down to the internet provider in the area
doesn't even know what their tech guys are actually doing.

[Comment 8 Robin Johnson 2005-08-08 10:06:28 UTC]

As a previous Telus user myself (hello from the Lower Mainland), I've 
definately found that with Telus, what they tell you they are doing, and what 
they actually do are very different - to the point of outright contradiction 
sometimes. I switched to Rogers (now Shaw) a long time ago because of this.

[Comment 9 Michael Hanselmann (hansmi) (RETIRED) 2005-08-18 12:12:32 UTC]

Closing