I cant get the this card to work in my server. Don't if this is the right place to ask for help. Here is a list of things i've observed: My Setup: Server Gentoo Linux with kernel version 3.14.0: Asus KGPE-D16 and S320-LP-CR driver: cxgb3 version: 1.1.5-ko firmware-version: T 7.12.0 TP 1.1.0 bus-info: 0000:02:00.0 Client Gentoo Linux with kernel version 3.14.0: Asus M5A97 EVO R2.0 and S320-LP-CR driver: cxgb3 version: 1.1.5-ko firmware-version: T 7.12.0 TP 1.1.0 bus-info: 0000:07:00.0 Server and Client are directly connect via sr multimode Fiber. Driver loads and find card on Server and Client. dmesg "modprobe cxgb3" sevrer: [ 44.920510] cxgb3: Chelsio T3 Network Driver - version 1.1.5-ko [ 45.256640] cxgb3 0000:02:00.0: irq 78 for MSI/MSI-X [ 45.256649] cxgb3 0000:02:00.0: irq 79 for MSI/MSI-X [ 45.256655] cxgb3 0000:02:00.0: irq 80 for MSI/MSI-X [ 45.256661] cxgb3 0000:02:00.0: irq 81 for MSI/MSI-X [ 45.256670] cxgb3 0000:02:00.0: irq 82 for MSI/MSI-X [ 45.256675] cxgb3 0000:02:00.0: irq 83 for MSI/MSI-X [ 45.256681] cxgb3 0000:02:00.0: irq 84 for MSI/MSI-X [ 45.256687] cxgb3 0000:02:00.0: irq 85 for MSI/MSI-X [ 45.256692] cxgb3 0000:02:00.0: irq 86 for MSI/MSI-X [ 45.256715] cxgb3 0000:02:00.0: Port 0 using 4 queue sets. [ 45.256717] cxgb3 0000:02:00.0: Port 1 using 4 queue sets. [ 45.256723] cxgb3 0000:02:00.0 eth1: Chelsio T320 10GBASE-R RNIC (rev 4) PCI Express x4 MSI-X [ 45.256725] cxgb3: eth1: 128MB CM, 256MB PMTX, 256MB PMRX, S/N: PT14100523 [ 45.256727] cxgb3 0000:02:00.0 eth2: Chelsio T320 10GBASE-R RNIC (rev 4) PCI Express x4 MSI-X dmesg "modprobe cxgb3" client: [ 80.501496] cxgb3: Chelsio T3 Network Driver - version 1.1.5-ko [ 80.845504] cxgb3 0000:07:00.0: irq 91 for MSI/MSI-X [ 80.845512] cxgb3 0000:07:00.0: irq 92 for MSI/MSI-X [ 80.845518] cxgb3 0000:07:00.0: irq 93 for MSI/MSI-X [ 80.845523] cxgb3 0000:07:00.0: irq 94 for MSI/MSI-X [ 80.845528] cxgb3 0000:07:00.0: irq 95 for MSI/MSI-X [ 80.845534] cxgb3 0000:07:00.0: irq 96 for MSI/MSI-X [ 80.845539] cxgb3 0000:07:00.0: irq 97 for MSI/MSI-X [ 80.845544] cxgb3 0000:07:00.0: irq 98 for MSI/MSI-X [ 80.845550] cxgb3 0000:07:00.0: irq 99 for MSI/MSI-X [ 80.845574] cxgb3 0000:07:00.0: Port 0 using 4 queue sets. [ 80.845575] cxgb3 0000:07:00.0: Port 1 using 4 queue sets. [ 80.845580] cxgb3 0000:07:00.0 eth1: Chelsio T320 10GBASE-R RNIC (rev 4) PCI Express x4 MSI-X [ 80.845581] cxgb3: eth1: 128MB CM, 256MB PMTX, 256MB PMRX, S/N: PT17100320 [ 80.845583] cxgb3 0000:07:00.0 eth2: Chelsio T320 10GBASE-R RNIC (rev 4) PCI Express x4 MSI-X server: ifconfig feth0 10.0.2.251 netmask 255.255.255.0 mtu 9000 up client: ifconfig feth0 10.0.2.1 netmask 255.255.255.0 mtu 9000 up ifconfig and ethtool server: feth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 inet 10.0.2.251 netmask 255.255.255.0 broadcast 10.0.2.255 inet6 fe80::207:43ff:fe06:7d0d prefixlen 64 scopeid 0x20<link> ether 00:07:43:06:7d:0d txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 40 memory 0xfbaff000-fbafffff Settings for feth0: Supported ports: [ AUI FIBRE ] Supported link modes: 10000baseT/Full 40000baseCR4/Full Supported pause frame use: No Supports auto-negotiation: No Advertised link modes: Not reported Advertised pause frame use: No Advertised auto-negotiation: No Speed: 10000Mb/s Duplex: Full Port: FIBRE PHYAD: 0 Transceiver: external Auto-negotiation: off Supports Wake-on: d Wake-on: d Current message level: 0x000000ff (255) drv probe link timer ifdown ifup rx_err tx_err Link detected: yes ifconfig and ethtool client: eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 inet 10.0.2.1 netmask 255.255.255.0 broadcast 10.0.2.255 inet6 fe80::207:43ff:fe06:8b79 prefixlen 64 scopeid 0x20<link> ether 00:07:43:06:8b:79 txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 8 bytes 680 (680.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 20 memory 0xfe081000-fe081fff Settings for eth1: Supported ports: [ AUI FIBRE ] Supported link modes: 10000baseT/Full 40000baseCR4/Full Supported pause frame use: No Supports auto-negotiation: No Advertised link modes: Not reported Advertised pause frame use: No Advertised auto-negotiation: No Speed: 10000Mb/s Duplex: Full Port: FIBRE PHYAD: 0 Transceiver: external Auto-negotiation: off Supports Wake-on: d Wake-on: d Current message level: 0x000000ff (255) drv probe link timer ifdown ifup rx_err tx_err Link detected: yes when i ping the client from server: ping 10.0.2.1 PING 10.0.2.1 (10.0.2.1) 56(84) bytes of data. From 10.0.2.251: icmp_seq=1 Destination Host Unreachable From 10.0.2.251: icmp_seq=2 Destination Host Unreachable From 10.0.2.251: icmp_seq=3 Destination Host Unreachable ^C --- 10.0.2.1 ping statistics --- 5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 4003ms server: feth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 inet 10.0.2.251 netmask 255.255.255.0 broadcast 10.0.2.255 inet6 fe80::207:43ff:fe06:7d0d prefixlen 64 scopeid 0x20<link> ether 00:07:43:06:7d:0d txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 40 memory 0xfbaff000-fbafffff client: eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 inet 10.0.2.1 netmask 255.255.255.0 broadcast 10.0.2.255 inet6 fe80::207:43ff:fe06:8b79 prefixlen 64 scopeid 0x20<link> ether 00:07:43:06:8b:79 txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 8 bytes 680 (680.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 20 memory 0xfe081000-fe081fff no packets counted, nothing seem to be send or received. when i ping the server from client: cube ~ # ping 10.0.2.251 PING 10.0.2.251 (10.0.2.251) 56(84) bytes of data. From 10.0.2.1: icmp_seq=1 Destination Host Unreachable From 10.0.2.1: icmp_seq=2 Destination Host Unreachable From 10.0.2.1: icmp_seq=3 Destination Host Unreachable ^C --- 10.0.2.251 ping statistics --- 4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2999ms server: feth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 inet 10.0.2.251 netmask 255.255.255.0 broadcast 10.0.2.255 inet6 fe80::207:43ff:fe06:7d0d prefixlen 64 scopeid 0x20<link> ether 00:07:43:06:7d:0d txqueuelen 1000 (Ethernet) RX packets 6 bytes 384 (384.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 40 memory 0xfbaff000-fbafffff client: eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 inet 10.0.2.1 netmask 255.255.255.0 broadcast 10.0.2.255 inet6 fe80::207:43ff:fe06:8b79 prefixlen 64 scopeid 0x20<link> ether 00:07:43:06:8b:79 txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 14 bytes 1064 (1.0 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 20 memory 0xfe081000-fe081fff arp request generted by client seem to be send and received. When i swap cards including it's transceiver modules, the problem did not swap too. Client can send Server can receive. So lwl and transceiver modules seem to be fine. on client side i can load the ifniband driver fine modprobe iw_cxgb3 [73482.006754] iw_cxgb3: Chelsio T3 RDMA Driver - version 1.1 [73482.014585] iw_cxgb3: Initialized device 0000:07:00.0 on the server side it fails: [73695.971785] iw_cxgb3: open_rnic_dev t3cdev ffff88040cc20000 [73695.971788] iw_cxgb3: Chelsio T3 RDMA Driver - version 1.1 [73695.971790] iw_cxgb3: cxio_rdev_open opening rnic dev ofld_dev0 [73695.971834] iw_cxgb3: cxio_rdev_open rnic ofld_dev0 info: tpt_base 0xa000000 tpt_top 0xbffffff num stags 32768 pbl_base 0xe000000 pbl_top 0xfffffff rqt_base 0xc000000, rqt_top 0xdffffff [73695.971837] iw_cxgb3: udbell_len 0x800000 udbell_physbase 0xfb000000 kdb_addr ffffc90015c14004 qpshift 7 qpnr 2048 qpmask 0x1f [73695.971853] iw_cxgb3: CtrlQP dma_addr 0x383988000 workq ffff880383988000 size 256 [73695.971855] cxio_rdev_open error 2 initializing ctrl_qp. [73695.971856] iw_cxgb3: Unable to open CXIO rdev I also noticed that on server there are no interupts for the card when i look into /proc/interupts on client there are as expected interupts counted any ideas where to look next ? Thanks in advance, Eric Grüttefien
(In reply to Eric Grüttefien from comment #0) > I cant get the this card to work in my server. That's a pity. > Don't if this is the right place to ask for help. It definitely isn't. This is a bug tracker. If you find a bug, you report it here. > any ideas where to look next ? We have web forums, IRC channels, mailing lists and more. See URL for more information. > Thanks in advance, You're welcome.
A fine loading driver wich updates the cards firmware, an ifup enables the link, but no interupts are generated but packet are received, then it must be a bug or am i wrong ? i'm not searching for a how to configure it right or somthing like that, i search help for narrowing the actual bug and where to report it.
(In reply to Eric Grüttefien from comment #2) > A fine loading driver wich updates the cards firmware, an ifup enables the > link, but no interupts are generated but packet are received, then it must > be a bug or am i wrong ? Sure. What is the bug, then? All I see is a report that your network interface isn't working. > i'm not searching for a how to configure it right or somthing like that, i > search help for narrowing the actual bug and where to report it. The usual way is to do the narrowing first and only then report your findings. Try the forums, IRC, mailing lists, or Google it. Also, don't reopen this bug report until you've found the bug.
