I have an interesting issue. I have an amd64 box running as a firewall/Server. It has most of the common services running. Sendmail, Apache, FTP, named etc.. I can ping anywhere from that box. All services work just super as one would expect. Except, I cannot traceroute out of that box. I started Wireshark and looked at the packets and they all have bad checksums, i.e.: Source port: 62654 (62654) Destination port: 33438 (33438) Length: 26 Checksum: 0xab6 [incorrect, should be 0x8b75] Every packet has the same length (26) and with the bad checksum, are being silently dropped. On one of my other firewall/Server boxes (Athlon XP 2700+ [32 bit]), all checks good, and all the packets are 18 in length with correct checksums. Emerge Info: [09:50][root@gateway:etc]# emerge --info Portage 2.1.1-r2 (default-linux/amd64/2006.1, gcc-4.1.1, glibc-2.4-r4, 2.6.17-gentoo-r5-Prod-2 x86_64) ================================================================= System uname: 2.6.17-gentoo-r5-Prod-2 x86_64 AMD Athlon(tm) 64 Processor 3500+ Gentoo Base System version 1.12.6 Last Sync: Fri, 22 Dec 2006 12:50:01 +0000 ccache version 2.3 [enabled] app-admin/eselect-compiler: [Not Present] dev-java/java-config: [Not Present] dev-lang/python: 2.4.3-r4 dev-python/pycrypto: 2.0.1-r5 dev-util/ccache: 2.3 dev-util/confcache: [Not Present] sys-apps/sandbox: 1.2.17 sys-devel/autoconf: 2.13, 2.60 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2 sys-devel/binutils: 2.16.1-r3 sys-devel/gcc-config: 1.3.14 sys-devel/libtool: 1.5.22 virtual/os-headers: 2.6.11-r2 ACCEPT_KEYWORDS="amd64" AUTOCLEAN="yes" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=k8 -O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/share/X11/app-defaults /usr/share/X11/xkb /usr/share/config" CONFIG_PROTECT_MASK="/etc/env.d /etc/gconf /etc/revdep-rebuild /etc/terminfo" CXXFLAGS="-march=k8 -O2 -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="autoconfig ccache distlocks metadata-transfer sandbox sfperms strict" GENTOO_MIRRORS="http://mirror.datapipe.net/gentoo http://gentoo.osuosl.org/ ftp://mirrors.tds.net/gentoo" MAKEOPTS="-j2" PKGDIR="/usr/portage/packages" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --delete-after --stats --timeout=180 --exclude='/distfiles' --exclude='/local' --exclude='/packages'" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" SYNC="rsync://sync.oneeighty.com/gentoo-portage" USE="amd64 X alsa_pcm_plugins_adpcm alsa_pcm_plugins_alaw alsa_pcm_plugins_asym alsa_pcm_plugins_copy alsa_pcm_plugins_dmix alsa_pcm_plugins_dshare alsa_pcm_plugins_dsnoop alsa_pcm_plugins_empty alsa_pcm_plugins_extplug alsa_pcm_plugins_file alsa_pcm_plugins_hooks alsa_pcm_plugins_iec958 alsa_pcm_plugins_ioplug alsa_pcm_plugins_ladspa alsa_pcm_plugins_lfloat alsa_pcm_plugins_linear alsa_pcm_plugins_meter alsa_pcm_plugins_mulaw alsa_pcm_plugins_multi alsa_pcm_plugins_null alsa_pcm_plugins_plug alsa_pcm_plugins_rate alsa_pcm_plugins_route alsa_pcm_plugins_share alsa_pcm_plugins_shm alsa_pcm_plugins_softvol berkdb caps clearpasswd cli cracklib crypt cups dlloader dri elibc_glibc fortran gdbm gpm gtk gtk+ iconv input_devices_evdev input_devices_keyboard input_devices_mouse isdnlog kde kernel_linux libg++ mbox ncurses nls nptl nptlonly pam pcre pdf perl ppds pppd python readline reflection session spl ssl tcpd udev unicode userland_GNU userlocales video_cards_apm video_cards_ark video_cards_ati video_cards_chips video_cards_cirrus video_cards_cyrix video_cards_dummy video_cards_fbdev video_cards_glint video_cards_i128 video_cards_i810 video_cards_mga video_cards_neomagic video_cards_nv video_cards_rendition video_cards_s3 video_cards_s3virge video_cards_savage video_cards_siliconmotion video_cards_sis video_cards_sisusb video_cards_tdfx video_cards_tga video_cards_trident video_cards_tseng video_cards_v4l video_cards_vesa video_cards_vga video_cards_via video_cards_vmware video_cards_voodoo xinetd xorg zlib" Unset: CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, LDFLAGS, LINGUAS, PORTAGE_RSYNC_EXTRA_OPTS, PORTDIR_OVERLAY lspci: 00:08.0 Ethernet controller: Intel Corporation 82557/8/9 [Ethernet Pro 100] (rev 08) 00:12.0 Ethernet controller: VIA Technologies, Inc. VT6102 [Rhine-II] (rev 78) dmesg regarding NIC's & Drivers Intel(R) PRO/1000 Network Driver - version 7.1.9-k4 Copyright (c) 1999-2006 Intel Corporation. e100: Intel(R) PRO/100 Network Driver, 3.5.10-k2-NAPI e100: Copyright(c) 1999-2005 Intel Corporation e100: eth0: e100_probe: addr 0xfa200000, irq 177, MAC addr 00:D0:B7:47:8F:48 via-rhine.c:v1.10-LK1.2.0-2.6 June-10-2004 Written by Donald Becker eth1: VIA Rhine II at 0x1ec00, 00:11:5b:f0:94:b0, IRQ 185. eth1: MII PHY found at address 1, status 0x786d advertising 05e1 Link 45e1 Hope this helps...
Reproducible on the latest kernel, currently 2.6.19?
(In reply to comment #1) > Reproducible on the latest kernel, currently 2.6.19? > Yes, even though I had to over ride the ~amd64. Exactly the same. Linux athlon 2.6.19-gentoo-r2-Prod-2 #1 SMP Mon Dec 25 19:46:47 MST 2006 x86_64 AMD Athlon(tm) 64 Processor 3500+ AuthenticAMD GNU/Linux
I'm having the same problem. I added some debugging code to traceroute; traceroute is calculating different checksums for each packets, but by the time tcpdump displays them and they go across the network, they all have the same invalid checksum. I have noted: 1. Disabling IP (not UDP!) checksums by passing "-x" to traceroute fixes the problem. 2. Specifying an explicit source address by passing "-s 12.34.56.78" to traceroute fixes the problem. 3. Specifying an interface to get the source address from by passing "-i eth1" to traceroute does not fix the problem. 4. Changing the address assigned to the interface on which the packets are being sent changes the invalid checksum. From this, I would guess that when traceroute allows the kernel to select the source address, it accidentally overwrites the checksum field. My guess is that some function in the kernel accesses part of the address as a native type such as int or long, and on a 64-bit system the variable is larger than expected. But I'm just a lowly user. Perhaps we should pass this to the kernel folks if it is, indeed, a real bug? By the way, packets sent by netcat have valid checksums, and packets sent by traceroute-nanog have no checksum, so those programs both work. Maybe comparing the ways they send UDP packets would reveal what traceroute is doing wrong itself and/or what traceroute is doing to trigger a kernel bug.
thanks for the follow up info. You'd think they would pass this up the chain. I was beginning to think I was the only one with this issue... Me a lowly user too ;)
comment #3 seems to suggest that only traceroute frames have incorrect checksums, whereas comment #0 seems to be talking about ALL network traffic. Is this correct? If so, these are 2 separate bugs.
RE: Comment 0. From what I can see, traceroute is the only net program I have had difficulty with. Everything else seems peachy keen... BTW, traceroute -x host.domainname.com also works on my machine as it does not generate the checksums...
I confirm this behavior on different platform with gentoo kernel x86 kernel gentoo 2.6.18-r3 x86 kernel gentoo 2.6.19-r2 but work on amd64 with gentoo kernel 2.6.18-gentoo-r2 each machine have different kind of network interface Also the strange thing i see with this is that kernel netfilter doesn't honor snat rules for this kind of packet. I have his rules on firewall /sbin/iptables -A POSTROUTING -t nat -o $PUB_DEV -s $INTERNAL_NET -j SNAT --to-source $PUB_IP rules is correctly done for traceroute with -x or -s argument but not for normal traceroute
just to clarify my previous comment. When i say that netfilter snat is not honored i mean that the packet is send on the output interface with address from internal net.
one of my colleague removed the following patch from the ebuild epatch ${FILESDIR}/${PN}-1.4a12-let_kernel_find_address.patch i did the same and traceroute work correctly. So the problem can be the patch or a kernel bug.
Eldad, any thoughts on this?
This is a traceroute bug introduced by a patch from bug #131723 Carlos and others tracked this down on IRC: traceroute is generating UDP checksums with the sender IP address of zero, then expecting the kernel to fill in the sender address thus making the checksum invalid
Ahh, makes sense. it's sending packets using the raw socket. I'll have a look. Daniel, perhaps we should retract the let-kernel-assign-address patch until I can fix this proper?
alright, I got a fix. This will disable checksumming when there is no address specified. the patch is here: http://dev.gentoo.org/~eldad/patches/26_all_traceroute-1.4a12-let_kernel_find_address_chksum.patch
Created attachment 129097 [details] traceroute-1.4_p12-r6.ebuild (patched) Test this ebuild and see if it fixes it.
The is a *new* package net-analyzer/traceroute-2.0.8-r2. Thank Mike Frysinger for digging up http://archives.gentoo.org/gentoo-dev/msg_144535.xml this package and pushing it into the tree. It should fix this issue. The only problem I see is the following note in net-analyzer/traceroute ChangeLog: 05 Sep 2007; Roy Marples <uberlord AT gentoo.org> traceroute-2.0.8.ebuild: Version 2 looks like it's Linux only, so FreeBSD keywords are dropped. But do we have this bug in FreeBSD? Will applied patch fix the issue? Roy, could you tell us, please?
Cool Beans. traceroute-2.0.8-r2 fixed it on my amd64 X2 boxes and one amd64 single core. Thanks so much... John
(In reply to comment #15) > But do we have this bug in FreeBSD? Will applied patch fix the issue? Roy, > could you tell us, please? Sorry for the delay, had big trouble getting wireshark working. Anyway, it's working now and the result is ..... unsure. Whilst unpatched it is generating an invalid checksum, with the patch outbound packets have NO checksum which I'm sure isn't optimal either.
FreeBSD traceroute does the same thing, so I'd say the patch is good for us too.
Finally this bug is FIXED. Thank you Eldad for the fix, Roy for tests and all others who reported / worked here!
