Bugzilla Bug 165784

I have recently upgraded to amavisd-new 2.4.5. This process has
also upgraded a lot of underlying perl packages on my system.

The problem that I'm seeing now is that special characters (iso8859-2 encoded)
in mail that goes through amavisd-new have started to be escaped in a strange
way, wreaking havoc in my terminal window when accessed with mail client.

Disabling amavisd-new as a content filter for postfix stops that from
happening.

I believe this is not an issue with amavisd itself, as downgrading to version
2.4.4 did not help.

The packages that were recently upgraded on my system are:

./virtual/perl-DB_File-1.815/
./virtual/perl-File-Spec-3.24/
./virtual/perl-Test-Simple-0.67/
./perl-core/File-Spec-3.24/
./perl-core/Test-Simple-0.67/
./perl-core/DB_File-1.815/
./mail-filter/amavisd-new-2.4.5/
./dev-perl/yaml-0.62/
./dev-perl/ExtUtils-CBuilder-0.18/
./dev-perl/extutils-parsexs-2.18/
./dev-perl/Archive-Zip-1.18/
./dev-perl/File-Which-0.05/
./dev-perl/module-build-0.28.06/

Do you have any idea what might be causing this erratic behaviour ?

emerge info:Portage 2.1.2-r8 (hardened/x86/2.6, gcc-3.4.6, glibc-2.3.6-r5,
2.6.19-hardened-r5 i686)
=================================================================
System uname: 2.6.19-hardened-r5 i686 Pentium II (Deschutes)
Gentoo Base System version 1.12.9
Timestamp of tree: Wed, 07 Feb 2007 10:50:01 +0000
distcc 2.18.3 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632)
[disabled]
dev-lang/python:     2.4.4
dev-python/pycrypto: 2.0.1-r5
sys-apps/sandbox:    1.2.18.1
sys-devel/autoconf:  2.60
sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10
sys-devel/binutils:  2.17
sys-devel/gcc-config: 1.3.14
sys-devel/libtool:   1.5.22
virtual/os-headers:  2.6.20
ACCEPT_KEYWORDS="x86 ~x86"
AUTOCLEAN="yes"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-s -pipe -march=pentium2 -O3 -funroll-loops -fstack-protector"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /var/bind"
CONFIG_PROTECT_MASK="/etc/env.d /etc/gconf /etc/revdep-rebuild /etc/terminfo"
CXXFLAGS="-s -pipe -march=pentium2 -O3 -funroll-loops -fstack-protector"
DISTDIR="/opt/distfiles"
FEATURES="autoconfig collision-protect distlocks metadata-transfer sandbox
sfperms strict userpriv usersandbox"
GENTOO_MIRRORS="http://linux.rz.ruhr-uni-bochum.de/download/gentoo-mirror
http://gentoo.mirror.solnet.ch http://trumpetti.atm.tut.fi/gentoo/"
MAKEOPTS="-j2"
PKGDIR="/usr/portage/packages"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress
--force --whole-file --delete --delete-after --stats --timeout=180
--exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/opt/portagetmp"
PORTDIR="/usr/portage"
SYNC="rsync://rsync5.pl.gentoo.org/gentoo-portage"
USE="bashlogger berkdb bzip2 caps chroot clearpasswd crypt dlloader elf ftp
glibc-omitfp hardened hpn mbox midi minimal ncurses nptl nptlonly pam
pam_chroot pam_timestamp pic png pwdb readline sendfile sftplogging symlink
tcpd threads userlocales x86 xinetd xorg zlib" ALSA_PCM_PLUGINS="adpcm alaw
asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa
lfloat linear meter mulaw multi null plug rate route share shm softvol"
ELIBC="glibc" INPUT_DEVICES="mouse keyboard" KERNEL="linux" LCD_DEVICES="bayrad
cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" USERLAND="GNU"
Unset:  CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, LDFLAGS,
LINGUAS, PORTAGE_RSYNC_EXTRA_OPTS, PORTDIR_OVERLAY

Can you provide a sample of e-mail before and after it passes amavisd-new ?

all right, the content of the email sent is this:

łąką szło dżdżownic parę

after going through amavisd becomes:

³¹k¹ sz³o d¿d¿ownic parê 

hope it helps and let me know if you need anything else

thank you

I was hoping for source files for those mails, but from these samples it
appears as if the text got converted to unicode somewhere along the way. I
assume that encoding is stated as ISO-8859-2 in e-mail headers, so the text is
being displayed incorrectly.

(In reply to comment #3)
> I was hoping for source files for those mails, but from these samples it
> appears as if the text got converted to unicode somewhere along the way. I
> assume that encoding is stated as ISO-8859-2 in e-mail headers, so the text is
> being displayed incorrectly.
> 
My test case is this:

- disable content filter (amavisd) in postfix, reload
- create plain text file with iso8859-2 chars in it
- mail it with 'mail' 
- in the email received - no special encoding headers, but the iso8859-2 chars
appear correctly in mail/pine/squirrelmail etc

- enable content filter, reload
- follow the above steps
- the email received has additional characters which escape iso8859-2
characters and are not interpreted correctly by neither of the above mail
programs.

I could have provided you with the emails directly from postfix queue but their
content is a little bit scrambled as far as I can say and guess they would be
useless in this case.

Hope it clears the issue a little bit.

(In reply to comment #4)
> I could have provided you with the emails directly from postfix queue but their
> content is a little bit scrambled as far as I can say and guess they would be
> useless in this case.

You can use postcat utility to dump mails inside the postfix queue to stdout in
human-readable form.

I'm interested in such mail before and after amavisd-new cripples it. The
"before" mail can be taken from sent folder in the e-mail client you used to
send it, if that's easier for you to obtain. Same for "after" mail, which can
be taken from inbox of the client you use to read it (or directly from mail
backend, such as maildir, whatever's more convenient). These are needed in
order to push this upstream - so they can compare the two e-mails and see what
exactly  changed.

> Hope it clears the issue a little bit.

every day you learn something new :)

here it is (output from postcat):

*** ENVELOPE RECORDS AF1E74045 ***
message_size:             365             199               1               0
content_filter: smtp-amavis:[127.0.0.1]:10024
message_arrival_time: Fri Feb  9 15:24:54 2007
create_time: Fri Feb  9 15:24:54 2007
named_attribute: rewrite_context=local
sender_fullname: Rewt Of All Evil
sender: XXXXXXXX@ibt.com.pl
*** MESSAGE CONTENTS AF1E74045 ***
Received: by felix.ibt.com.pl (Postfix, from userid 1000)
        id AF1E74045; Fri,  9 Feb 2007 15:24:54 +0100 (CET)
Subject: testmail
To: <XXXXXX@ibt.com.pl>
X-Mailer: mail (GNU Mailutils 0.6)
Message-Id: <20070209142454.AF1E74045@felix.ibt.com.pl>
Date: Fri,  9 Feb 2007 15:24:54 +0100 (CET)
From: XXXXX@ibt.com.pl (Rewt Of All Evil)

łąką szło dżdżownic parę
*** HEADER EXTRACTED AF1E74045 ***
original_recipient: XXXXX@ibt.com.pl
recipient: XXXX@ibt.com.pl
*** MESSAGE FILE END AF1E74045 ***

and after being passed through amavisd & delivered through postfix/procmail:

From XXXXX@ibt.com.pl  Fri Feb  9 15:30:22 2007
Return-Path: <XXXX@ibt.com.pl>
X-Original-To: xxxx@ibt.com.pl
Delivered-To: xxxx@ibt.com.pl
Received: from localhost (localhost [127.0.0.1])
        by felix.ibt.com.pl (Postfix) with ESMTP id B29574045
        for <XXXX@ibt.com.pl>; Fri,  9 Feb 2007 15:30:22 +0100 (CET)
X-Virus-Scanned: amavisd-new at ibt.com.pl
Received: from felix.ibt.com.pl ([127.0.0.1])
        by localhost (felix.ibt.com.pl [127.0.0.1]) (amavisd-new, port 10024)
        with LMTP id 3SS14OWPCQcc for <XXXX@ibt.com.pl>;
        Fri,  9 Feb 2007 15:30:22 +0100 (CET)
Received: by felix.ibt.com.pl (Postfix, from userid 1000)
        id 7A3144053; Fri,  9 Feb 2007 15:30:22 +0100 (CET)
Subject: testmail
To: <XXXX@ibt.com.pl>
X-Mailer: mail (GNU Mailutils 0.6)
Message-Id: <20070209143022.7A3144053@felix.ibt.com.pl>
Date: Fri,  9 Feb 2007 15:30:22 +0100 (CET)
From: XXX@ibt.com.pl (Rewt Of All Evil)

³¹k¹ sz³o d¿d¿ownic parê





thanks!

I've done some more googling and it seems
that I am not the one affected by the problem after all :)

The good news is that the culprit of the problem has been found.

It's libnet version 1.20 which is buggy.

http://marc.theaimsgroup.com/?l=amavis-user&m=117095734709511&w=2

hope it helps someone out there!

cheers

(In reply to comment #7)
> I've done some more googling and it seems
> that I am not the one affected by the problem after all :)
> 
> The good news is that the culprit of the problem has been found.
> 
> It's libnet version 1.20 which is buggy.
> 
> http://marc.theaimsgroup.com/?l=amavis-user&m=117095734709511&w=2
> 
> hope it helps someone out there!
> 
> cheers
> 

Thanks for the research, I admittedly did not have time to investigate it.

Reassigning to perl people, so they can react - there is no bug reported in our
bugzilla for this yet, and the patch is not applied to libnet-1.2.0 in the
tree.

There is also no progress on upstream bug
(http://rt.cpan.org/Public/Bug/Display.html?id=24835), apart from the suggested
patch.

1.20-r1 added with patch, 1.20 itself removed. Thanks barthek :)