165784 2007-02-07 16:39 0000 perl-core/libnet-1.2.0 - problem with iso8859-2 chars 2007-03-25 12:07:05 0000 1 1 1 Unclassified Gentoo Linux Ebuilds unspecified All Linux RESOLVED FIXED P2 normal --- 1 gejzer@ibt.com.pl perl@gentoo.org antivirus@gentoo.org net-mail@gentoo.org gejzer@ibt.com.pl 2007-02-07 16:39:01 0000 I have recently upgraded to amavisd-new 2.4.5. This process has also upgraded a lot of underlying perl packages on my system. The problem that I'm seeing now is that special characters (iso8859-2 encoded) in mail that goes through amavisd-new have started to be escaped in a strange way, wreaking havoc in my terminal window when accessed with mail client. Disabling amavisd-new as a content filter for postfix stops that from happening. I believe this is not an issue with amavisd itself, as downgrading to version 2.4.4 did not help. The packages that were recently upgraded on my system are: ./virtual/perl-DB_File-1.815/ ./virtual/perl-File-Spec-3.24/ ./virtual/perl-Test-Simple-0.67/ ./perl-core/File-Spec-3.24/ ./perl-core/Test-Simple-0.67/ ./perl-core/DB_File-1.815/ ./mail-filter/amavisd-new-2.4.5/ ./dev-perl/yaml-0.62/ ./dev-perl/ExtUtils-CBuilder-0.18/ ./dev-perl/extutils-parsexs-2.18/ ./dev-perl/Archive-Zip-1.18/ ./dev-perl/File-Which-0.05/ ./dev-perl/module-build-0.28.06/ Do you have any idea what might be causing this erratic behaviour ? emerge info:Portage 2.1.2-r8 (hardened/x86/2.6, gcc-3.4.6, glibc-2.3.6-r5, 2.6.19-hardened-r5 i686) ================================================================= System uname: 2.6.19-hardened-r5 i686 Pentium II (Deschutes) Gentoo Base System version 1.12.9 Timestamp of tree: Wed, 07 Feb 2007 10:50:01 +0000 distcc 2.18.3 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632) [disabled] dev-lang/python: 2.4.4 dev-python/pycrypto: 2.0.1-r5 sys-apps/sandbox: 1.2.18.1 sys-devel/autoconf: 2.60 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10 sys-devel/binutils: 2.17 sys-devel/gcc-config: 1.3.14 sys-devel/libtool: 1.5.22 virtual/os-headers: 2.6.20 ACCEPT_KEYWORDS="x86 ~x86" AUTOCLEAN="yes" CBUILD="i686-pc-linux-gnu" CFLAGS="-s -pipe -march=pentium2 -O3 -funroll-loops -fstack-protector" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /var/bind" CONFIG_PROTECT_MASK="/etc/env.d /etc/gconf /etc/revdep-rebuild /etc/terminfo" CXXFLAGS="-s -pipe -march=pentium2 -O3 -funroll-loops -fstack-protector" DISTDIR="/opt/distfiles" FEATURES="autoconfig collision-protect distlocks metadata-transfer sandbox sfperms strict userpriv usersandbox" GENTOO_MIRRORS="http://linux.rz.ruhr-uni-bochum.de/download/gentoo-mirror http://gentoo.mirror.solnet.ch http://trumpetti.atm.tut.fi/gentoo/" MAKEOPTS="-j2" PKGDIR="/usr/portage/packages" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --delete-after --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/opt/portagetmp" PORTDIR="/usr/portage" SYNC="rsync://rsync5.pl.gentoo.org/gentoo-portage" USE="bashlogger berkdb bzip2 caps chroot clearpasswd crypt dlloader elf ftp glibc-omitfp hardened hpn mbox midi minimal ncurses nptl nptlonly pam pam_chroot pam_timestamp pic png pwdb readline sendfile sftplogging symlink tcpd threads userlocales x86 xinetd xorg zlib" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mulaw multi null plug rate route share shm softvol" ELIBC="glibc" INPUT_DEVICES="mouse keyboard" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" USERLAND="GNU" Unset: CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, LDFLAGS, LINGUAS, PORTAGE_RSYNC_EXTRA_OPTS, PORTDIR_OVERLAY ticho@gentoo.org 2007-02-07 23:36:06 0000 Can you provide a sample of e-mail before and after it passes amavisd-new ? gejzer@ibt.com.pl 2007-02-08 09:59:02 0000 all right, the content of the email sent is this: łąką szło dżdżownic parę after going through amavisd becomes: ³¹kÂą szÂło dÂżdÂżownic parĂŞ hope it helps and let me know if you need anything else thank you ticho@gentoo.org 2007-02-09 09:18:47 0000 I was hoping for source files for those mails, but from these samples it appears as if the text got converted to unicode somewhere along the way. I assume that encoding is stated as ISO-8859-2 in e-mail headers, so the text is being displayed incorrectly. gejzer@ibt.com.pl 2007-02-09 09:49:03 0000 (In reply to comment #3) > I was hoping for source files for those mails, but from these samples it > appears as if the text got converted to unicode somewhere along the way. I > assume that encoding is stated as ISO-8859-2 in e-mail headers, so the text is > being displayed incorrectly. > My test case is this: - disable content filter (amavisd) in postfix, reload - create plain text file with iso8859-2 chars in it - mail it with 'mail' - in the email received - no special encoding headers, but the iso8859-2 chars appear correctly in mail/pine/squirrelmail etc - enable content filter, reload - follow the above steps - the email received has additional characters which escape iso8859-2 characters and are not interpreted correctly by neither of the above mail programs. I could have provided you with the emails directly from postfix queue but their content is a little bit scrambled as far as I can say and guess they would be useless in this case. Hope it clears the issue a little bit. ticho@gentoo.org 2007-02-09 14:16:04 0000 (In reply to comment #4) > I could have provided you with the emails directly from postfix queue but their > content is a little bit scrambled as far as I can say and guess they would be > useless in this case. You can use postcat utility to dump mails inside the postfix queue to stdout in human-readable form. I'm interested in such mail before and after amavisd-new cripples it. The "before" mail can be taken from sent folder in the e-mail client you used to send it, if that's easier for you to obtain. Same for "after" mail, which can be taken from inbox of the client you use to read it (or directly from mail backend, such as maildir, whatever's more convenient). These are needed in order to push this upstream - so they can compare the two e-mails and see what exactly changed. > Hope it clears the issue a little bit. gejzer@ibt.com.pl 2007-02-09 14:32:15 0000 every day you learn something new :) here it is (output from postcat): *** ENVELOPE RECORDS AF1E74045 *** message_size: 365 199 1 0 content_filter: smtp-amavis:[127.0.0.1]:10024 message_arrival_time: Fri Feb 9 15:24:54 2007 create_time: Fri Feb 9 15:24:54 2007 named_attribute: rewrite_context=local sender_fullname: Rewt Of All Evil sender: XXXXXXXX@ibt.com.pl *** MESSAGE CONTENTS AF1E74045 *** Received: by felix.ibt.com.pl (Postfix, from userid 1000) id AF1E74045; Fri, 9 Feb 2007 15:24:54 +0100 (CET) Subject: testmail To: <XXXXXX@ibt.com.pl> X-Mailer: mail (GNU Mailutils 0.6) Message-Id: <20070209142454.AF1E74045@felix.ibt.com.pl> Date: Fri, 9 Feb 2007 15:24:54 +0100 (CET) From: XXXXX@ibt.com.pl (Rewt Of All Evil) łąką szło dżdżownic parę *** HEADER EXTRACTED AF1E74045 *** original_recipient: XXXXX@ibt.com.pl recipient: XXXX@ibt.com.pl *** MESSAGE FILE END AF1E74045 *** and after being passed through amavisd & delivered through postfix/procmail: From XXXXX@ibt.com.pl Fri Feb 9 15:30:22 2007 Return-Path: <XXXX@ibt.com.pl> X-Original-To: xxxx@ibt.com.pl Delivered-To: xxxx@ibt.com.pl Received: from localhost (localhost [127.0.0.1]) by felix.ibt.com.pl (Postfix) with ESMTP id B29574045 for <XXXX@ibt.com.pl>; Fri, 9 Feb 2007 15:30:22 +0100 (CET) X-Virus-Scanned: amavisd-new at ibt.com.pl Received: from felix.ibt.com.pl ([127.0.0.1]) by localhost (felix.ibt.com.pl [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 3SS14OWPCQcc for <XXXX@ibt.com.pl>; Fri, 9 Feb 2007 15:30:22 +0100 (CET) Received: by felix.ibt.com.pl (Postfix, from userid 1000) id 7A3144053; Fri, 9 Feb 2007 15:30:22 +0100 (CET) Subject: testmail To: <XXXX@ibt.com.pl> X-Mailer: mail (GNU Mailutils 0.6) Message-Id: <20070209143022.7A3144053@felix.ibt.com.pl> Date: Fri, 9 Feb 2007 15:30:22 +0100 (CET) From: XXX@ibt.com.pl (Rewt Of All Evil) ³¹kÂą szÂło dÂżdÂżownic parĂŞ thanks! gejzer@ibt.com.pl 2007-02-13 10:59:33 0000 I've done some more googling and it seems that I am not the one affected by the problem after all :) The good news is that the culprit of the problem has been found. It's libnet version 1.20 which is buggy. http://marc.theaimsgroup.com/?l=amavis-user&m=117095734709511&w=2 hope it helps someone out there! cheers ticho@gentoo.org 2007-02-14 03:01:39 0000 (In reply to comment #7) > I've done some more googling and it seems > that I am not the one affected by the problem after all :) > > The good news is that the culprit of the problem has been found. > > It's libnet version 1.20 which is buggy. > > http://marc.theaimsgroup.com/?l=amavis-user&m=117095734709511&w=2 > > hope it helps someone out there! > > cheers > Thanks for the research, I admittedly did not have time to investigate it. Reassigning to perl people, so they can react - there is no bug reported in our bugzilla for this yet, and the patch is not applied to libnet-1.2.0 in the tree. There is also no progress on upstream bug (http://rt.cpan.org/Public/Bug/Display.html?id=24835), apart from the suggested patch. mcummings@gentoo.org 2007-03-25 12:07:05 0000 1.20-r1 added with patch, 1.20 itself removed. Thanks barthek :)