While booting, bonding tries to get the link speed before it's actually given. i.e. ADDRCONF(NETDEV_UP): bond0: link is not ready bonding: bond0: Warning: failed to get speed and duplex from eth1, assumed to be 100Mb/sec and Full. and THEN: tg3: eth0: Link is up at 1000 Mbps, full duplex. tg3: eth0: Flow control is on for TX and on for RX. tg3: eth1: Link is up at 1000 Mbps, full duplex. tg3: eth1: Flow control is on for TX and on for RX. Problem: My two NICs are gigabit, as you see above. As it doesn't know what the link speed is (since it isn't ready at the time) it falls back to default, assuming it to be 100Mb/sec. Partial dmesg: [ 11.785634] VFS: Mounted root (ext4 filesystem) readonly on device 8:3. [ 11.786243] Freeing unused kernel memory: 380k freed [ 11.786486] Write protecting the kernel text: 4492k [ 11.786814] Write protecting the kernel read-only data: 1852k [ 11.811576] khelper used greatest stack depth: 6904 bytes left [ 12.083425] consoletype used greatest stack depth: 6844 bytes left [ 12.101701] stty used greatest stack depth: 6492 bytes left [ 12.320281] cp used greatest stack depth: 6396 bytes left [ 12.468164] udev: starting version 141 [ 12.504548] usb usb1: uevent [ 12.504609] usb 1-0:1.0: uevent [ 12.504778] usb 1-2: uevent [ 12.504847] usb 1-2:1.0: uevent [ 12.579300] udev: renamed network interface eth1 to eth2 [ 12.581375] udev: renamed network interface eth0 to eth3 [ 12.605356] udev: renamed network interface eth2_rename to eth4 [ 12.626310] udev: renamed network interface eth3_rename to eth0 [ 12.628776] udev: renamed network interface eth4_rename to eth1 [ 12.643152] udevd used greatest stack depth: 6324 bytes left [ 13.152232] EXT4 FS on sda3, internal journal on sda3:8 [ 13.374194] Adding 1049592k swap on /dev/sda2. Priority:-1 extents:1 across:1049592k [ 15.479401] bash used greatest stack depth: 6304 bytes left [ 23.851838] ADDRCONF(NETDEV_UP): bond0: link is not ready [ 23.905889] tg3 0000:0a:01.0: PME# disabled [ 24.116415] ADDRCONF(NETDEV_UP): eth0: link is not ready [ 24.141413] tg3 0000:0a:02.0: PME# disabled [ 24.351907] ADDRCONF(NETDEV_UP): eth1: link is not ready [ 24.970021] tg3 0000:0a:01.0: PME# disabled [ 25.192773] bonding: bond0: Warning: failed to get speed and duplex from eth0, assumed to be 100Mb/sec and Full. [ 25.192807] bonding: bond0: enslaving eth0 as an active interface with an up link. [ 25.193016] ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready [ 25.800019] tg3 0000:0a:02.0: PME# disabled [ 26.022752] bonding: bond0: Warning: failed to get speed and duplex from eth1, assumed to be 100Mb/sec and Full. [ 26.022770] bonding: bond0: enslaving eth1 as an active interface with an up link. [ 28.424502] tg3: eth0: Link is up at 1000 Mbps, full duplex. [ 28.424508] tg3: eth0: Flow control is on for TX and on for RX. [ 29.027500] tg3: eth1: Link is up at 1000 Mbps, full duplex. [ 29.027504] tg3: eth1: Flow control is on for TX and on for RX. [ 36.080006] bond0: no IPv6 routers present Uh oh. Bonding is either requesting too early, or tg3 is reporting too late. Reproducible: Always Steps to Reproduce: 1. Set up bonding 2. Reboot 3. Check dmesg Actual Results: It isn't using the correct 1000 Mbps speed but rather the assumed 100 Mbps. Expected Results: Bonding should query the tg3 driver later, or the tg3 driver should report earlier. server2 ~ # emerge --info Portage 2.1.6.13 (default/linux/x86/2008.0, gcc-4.3.2, glibc-2.9_p20081201-r2, 2.6.29-gentoo-r5 i686) ================================================================= System uname: Linux-2.6.29-gentoo-r5-i686-Intel-R-_XEON-TM-_MP_CPU_2.00GHz-with-glibc2.0 Timestamp of tree: Mon, 29 Jun 2009 02:45:01 +0000 app-shells/bash: 3.2_p39 dev-lang/python: 2.5.4-r2 sys-apps/baselayout: 1.12.11.1 sys-apps/sandbox: 1.6-r2 sys-devel/autoconf: 2.63 sys-devel/automake: 1.7.9-r1, 1.9.6-r2, 1.10.2 sys-devel/binutils: 2.18-r3 sys-devel/gcc-config: 1.4.1 sys-devel/libtool: 1.5.26 virtual/os-headers: 2.6.27-r2 ACCEPT_KEYWORDS="x86" CBUILD="i686-pc-linux-gnu" CFLAGS="-O2 -march=i686 -pipe" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /var/bind" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/sandbox.d /etc/terminfo /etc/udev/rules.d" CXXFLAGS="-O2 -march=i686 -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="distlocks fixpackages parallel-fetch protect-owned sandbox sfperms strict unmerge-orphans userfetch" GENTOO_MIRRORS="http://debian.ihug.co.nz/gentoo/" LDFLAGS="-Wl,-O1" MAKEOPTS="-j9" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" SYNC="rsync://linux.gamers.net.nz/gentoo-portage" USE="acl berkdb bzip2 cli cracklib crypt cups dri fortran gdbm gpm iconv ipv6 isdnlog midi mudflap ncurses nls nptl nptlonly openmp pam pcre perl pppd python readline reflection session spl ssl sysfs tcpd unicode x86 xorg zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" APACHE2_MPMS="itk" ELIBC="glibc" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" USERLAND="GNU" VIDEO_CARDS="fbdev glint i810 intel mach64 mga neomagic nv r128 radeon savage sis tdfx trident vesa vga via vmware voodoo" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, LANG, LC_ALL, LINGUAS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PORTDIR_OVERLAY
Have you checked it on older kernels? Also what kind of switch is in use?
I do update kernels regularly, but I can't recall if bond0 was ever working at 1Gbps link speed before as I only noticed this error for the first time just now. The switch in use is a TP-Link TL-SG1016.
Please test with 2.6.28 and report if this is still a problem.
I can now confirm that the same problem is encountered in 2.6.28-r5: server2 ~ # uname -a Linux server2 2.6.28-gentoo-r5 #1 SMP Wed Jul 1 04:12:59 NZST 2009 i686 Intel(R) XEON(TM) MP CPU 2.00GHz GenuineIntel GNU/Linux [ 11.791655] VFS: Mounted root (ext4 filesystem) readonly. [ 11.792293] Freeing unused kernel memory: 364k freed [ 11.792537] Write protecting the kernel text: 4348k [ 11.792807] Write protecting the kernel read-only data: 1764k [ 11.814405] khelper used greatest stack depth: 6616 bytes left [ 12.165832] stty used greatest stack depth: 6352 bytes left [ 12.395194] cp used greatest stack depth: 6228 bytes left [ 12.591033] udev: starting version 141 [ 12.630549] usb usb1: uevent [ 12.630620] usb 1-0:1.0: uevent [ 12.630741] usb 1-2: uevent [ 12.630799] usb 1-2:1.0: uevent [ 12.699551] udev: renamed network interface eth0 to eth3 [ 12.701815] udev: renamed network interface eth1 to eth2 [ 12.739651] udev: renamed network interface eth2_rename to eth4 [ 12.747003] udev: renamed network interface eth3_rename to eth0 [ 12.750273] udev: renamed network interface eth4_rename to eth1 [ 13.287010] EXT4 FS on sda3, internal journal on sda3:8 [ 13.325693] mount used greatest stack depth: 5852 bytes left [ 13.537037] Adding 1049592k swap on /dev/sda2. Priority:-1 extents:1 across:1049592k [ 21.285309] ADDRCONF(NETDEV_UP): bond0: link is not ready [ 21.555151] ADDRCONF(NETDEV_UP): eth0: link is not ready [ 21.793180] ADDRCONF(NETDEV_UP): eth1: link is not ready [ 22.634449] bonding: bond0: Warning: failed to get speed and duplex from eth0, assumed to be 100Mb/sec and Full. [ 22.634485] bonding: bond0: enslaving eth0 as an active interface with an up link. [ 22.635478] ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready [ 23.464499] bonding: bond0: Warning: failed to get speed and duplex from eth1, assumed to be 100Mb/sec and Full. [ 23.464516] bonding: bond0: enslaving eth1 as an active interface with an up link. [ 25.865496] tg3: eth0: Link is up at 1000 Mbps, full duplex. [ 25.865502] tg3: eth0: Flow control is on for TX and on for RX. [ 26.468499] tg3: eth1: Link is up at 1000 Mbps, full duplex. [ 26.468503] tg3: eth1: Flow control is on for TX and on for RX. [ 33.123017] bond0: no IPv6 routers present
I'd say this is normal and expected, however let kernel teem resolv this.
What bonding parameters are set in /etc/modules.autoload.d/kernel-2.6? Can I see /etc/conf.d/net also?
I never use modules, I always compile in the kernel - /etc/modules.autoload.d/kernel-2.6 has no uncommented lines. Here is my /etc/conf.d/net: # This blank configuration will automatically use DHCP for any net.* # scripts in /etc/init.d. To create a more complete configuration, # please review /etc/conf.d/net.example and save your configuration # in /etc/conf.d/net (this file :]!). dns_domain_bond0="mydomain.com" dns_servers_bond0="192.168.2.1" slaves_bond0="eth0 eth1" config_bond0=( "192.168.2.5 netmask 255.255.255.0 brd 192.168.2.255" ) routes_bond0=( "default via 192.168.2.1" ) config_eth0=( "null" ) config_eth1=( "null" ) Thanks.
Can you configure the driver as a module so we can pass some parameters as it is the only way to do so, and the recommended way to use the bonding driver in linux
Alright - I configured Network Bonding as a module [M] and now it seems to be working. (Why does it have to be a module?) In the meantime I also added another two NICs; now totalling four at 4Gbps. [ 11.899574] VFS: Mounted root (ext4 filesystem) readonly on device 8:3. [ 11.900198] Freeing unused kernel memory: 380k freed [ 11.900436] Write protecting the kernel text: 4492k [ 11.900761] Write protecting the kernel read-only data: 1852k [ 11.925058] khelper used greatest stack depth: 6652 bytes left [ 12.248287] stty used greatest stack depth: 6492 bytes left [ 12.469710] cp used greatest stack depth: 6396 bytes left [ 12.540104] udevadm used greatest stack depth: 6256 bytes left [ 12.669488] udev: starting version 141 [ 12.717497] usb usb1: uevent [ 12.717572] usb 1-0:1.0: uevent [ 12.717721] usb 1-2: uevent [ 12.717813] usb 1-2:1.0: uevent [ 13.377536] EXT4 FS on sda3, internal journal on sda3:8 [ 13.408575] mount used greatest stack depth: 5864 bytes left [ 13.566585] Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008) [ 13.566594] bonding: MII link monitoring set to 100 ms [ 13.675246] Adding 1049592k swap on /dev/sda2. Priority:-1 extents:1 across:1049592k [ 21.808655] ADDRCONF(NETDEV_UP): bond0: link is not ready [ 21.878331] ADDRCONF(NETDEV_UP): eth0: link is not ready [ 21.879378] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX [ 21.880109] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 21.901678] ADDRCONF(NETDEV_UP): eth1: link is not ready [ 21.902377] e1000: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX [ 21.902987] ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready [ 21.923567] tg3 0000:0a:01.0: PME# disabled [ 22.134080] ADDRCONF(NETDEV_UP): eth2: link is not ready [ 22.157554] tg3 0000:0a:02.0: PME# disabled [ 22.368056] ADDRCONF(NETDEV_UP): eth3: link is not ready [ 22.411142] bonding: bond0: enslaving eth0 as an active interface with a down link. [ 22.433998] bonding: bond0: enslaving eth1 as an active interface with a down link. [ 23.047020] tg3 0000:0a:01.0: PME# disabled [ 23.269713] bonding: bond0: enslaving eth2 as an active interface with a down link. [ 23.872018] tg3 0000:0a:02.0: PME# disabled [ 24.094717] bonding: bond0: enslaving eth3 as an active interface with a down link. [ 25.527370] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX [ 25.533015] bonding: bond0: link status definitely up for interface eth0. [ 25.533378] ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready [ 25.561366] e1000: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX [ 25.633015] bonding: bond0: link status definitely up for interface eth1. [ 26.175090] bond0: duplicate address detected! [ 26.175111] bond0: duplicate address detected! [ 26.496498] tg3: eth2: Link is up at 1000 Mbps, full duplex. [ 26.496503] tg3: eth2: Flow control is on for TX and on for RX. [ 26.533014] bonding: bond0: link status definitely up for interface eth2. [ 27.099502] tg3: eth3: Link is up at 1000 Mbps, full duplex. [ 27.099506] tg3: eth3: Flow control is on for TX and on for RX. [ 27.133026] bonding: bond0: link status definitely up for interface eth3. Is it still normal to have all the "link is not ready" messages? Two more questions: - Are the 'duplicate address detected' lines okay to ignore? They seem like errors to me. - What does "tg3 0000:0a:02.0: PME# disabled" mean exactly? Thank you very much!
The upstream developers recommend that you use this driver as a module since it's the only way to pass parameters to the driver or configure more than one bonding device. I believe PME is referring to PCI Express Power Management. There's a decent amount of documentation on this topic, feel free to google for more information. You dmesg looks good to me with 4 nic's reporting ready and available at 1000 Mbps.
