PS: sorry for my bad english, i french The xen network bridge not work because, the network for create bridge not stop correctely i have debuged the xend and /etc/xen/scripts/network-bridge script And here a result <<< before ifdown eth0 * Unmounting network filesystems ... [ ok ] * Stopping sshd ... [ !! ] * ERROR: problems stopping dependent services. * "net.eth0" is still up. after ifdown eth0 eth0 Link encap:Ethernet HWaddr 00:02:44:32:FE:BA inet addr:192.168.0.4 Bcast:192.168.0.255 Mask:255.255.255.0 inet6 addr: fe80::202:44ff:fe32:feba/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:59 errors:0 dropped:0 overruns:0 frame:0 TX packets:57 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:5408 (5.2 Kb) TX bytes:4936 (4.8 Kb) Interrupt:5 Base address:0xd000 >>> the eth0 interface not down, beacause error for ended sshd service (because not arealdy loaded) And after start the computer, if i restart the /etc/inid.d/xen restart, run correctely because the sshd have now started i resove this with edit a /etc/init.d/xend script Replace depend function by depend() { need net before xendomain ntpd nfs nfsmount rsyncd portmap after sshd } Reproducible: Always Steps to Reproduce: xen 3.0.0_pre20051027 xen-source 2.6.12.5-r2 emerge info result Portage 2.0.51.22-r2 (default-linux/x86/2005.0, gcc-3.3.6, glibc-2.3.5-r1, 2.6.12.6.5-xen i686) ================================================================= System uname: 2.6.12.6.5-xen i686 unknown Gentoo Base System version 1.6.13 ccache version 2.3 [disabled] dev-lang/python: 2.3.5-r2 sys-apps/sandbox: 1.2.12 sys-devel/autoconf: 2.13, 2.59-r6 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6 sys-devel/binutils: 2.15.92.0.2-r10 sys-devel/libtool: 1.5.18-r1 virtual/os-headers: 2.6.11-r2 ACCEPT_KEYWORDS="x86" AUTOCLEAN="yes" CBUILD="i686-pc-linux-gnu" CFLAGS="-O2 -march=athlon-xp -fomit-frame-pointer" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3.4/env /usr/kde/3.4/share/config /usr/kde/3.4/shutdown /usr/kde/3/share/config /usr/lib/X11/xkb /usr/share/config /var/qmail/control" CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d" CXXFLAGS="-O2 -march=athlon-xp -fomit-frame-pointer" DISTDIR="/usr/portage/distfiles" FEATURES="autoconfig distlocks sandbox sfperms strict" GENTOO_MIRRORS="http://distfiles.gentoo.org http://distro.ibiblio.org/pub/Linux/distributions/gentoo" LANG="fr_FR@euro" LC_ALL="fr_FR@euro" LINGUAS="fr" PKGDIR="/usr/portage/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="x86 X alsa apm arts audiofile avi bash-completion berkdb bitmap-fonts cddb cdparanoia cdr clamav crypt cups curl dv eds emboss encode esd exif fam flac foomaticdb fortran gd gdbm gif gnome gphoto2 gpm gstreamer gtk gtk2 imagemagick imlib ipv6 java jpeg junit kde ldap libg++ libwww mad mikmod mmx motif mp3 mpeg mppe-mppc mysql nas ncurses nls nvidia ogg oggvorbis opengl oss pam pdflib perl png ppds python qt quicktime readline samba sdl slang spell sqlite ssl stream svga symlink tcltk tcpd tiff truetype truetype-fonts type1-fonts udev vorbis win32codecs xine xml xml2 xv xvid zlib linguas_fr userland_GNU kernel_linux elibc_glibc" Unset: ASFLAGS, CTARGET, LDFLAGS, MAKEOPTS
network-bridge does ifdown ("/etc/init.d/net.eth0 stop") and then ifup ("/etc/init.d/net.eth0 start"). "net.eth0 stop" should bring down sshd and everything else. If /etc/init.d/xend runs after /etc/init.d/sshd then the xend init script will stop sshd, bring down eth0, bring it up again, but sshd doesn't get restarted, so I don't think that is a solution to the problem you're having.
you are right, I think the bug is in gentoo runscript ? Because in the sshd script they are line "need net", but xen trying stop for create network-bridge and the sshd not already started. and i didn't reason, ifdown (/etc/init.d/net.eth0 stop) try stop sshd (but not started, it started after xend script)
Why not create the bridge using Gentoo net scripts? http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=4&chap=3#doc_chap7
(In reply to comment #3) > Why not create the bridge using Gentoo net scripts? > http://www.gentoo.org/doc/en/handbook/handbook-x86.xml? part=4&chap=3#doc_chap7 because xend create bridge automaticaly with /etc/xen/scripts/network-bridge
(In reply to comment #4) > (In reply to comment #3) > > Why not create the bridge using Gentoo net scripts? > > http://www.gentoo.org/doc/en/handbook/handbook-x86.xml? > part=4&chap=3#doc_chap7 > > because xend create bridge automaticaly with /etc/xen/scripts/network-bridge So stop Xen from doing this and the issue is solved :) At least as far as I see it, it is.
It would be nice to have the networking more gentoo-like. However, take a look at the network-bridge script. op_start does more than just create a bridge, can the gentoo network init scripts handle the veth0/vif0 stuff easily, or is it just going to get more complicated, with user confusion over whether to use the xen or gentoo networking stuff added in. Also it's another thing to maintain since we couldn't rely on the xen guys fixes ending up in that script automatically.
(In reply to comment #6) > It would be nice to have the networking more gentoo-like. However, take a look > at the network-bridge script. op_start does more than just create a bridge, can > the gentoo network init scripts handle the veth0/vif0 stuff easily, or is it > just going to get more complicated, with user confusion over whether to use the > xen or gentoo networking stuff added in. Also it's another thing to maintain > since we couldn't rely on the xen guys fixes ending up in that script automatically. veth0/vif0 = virtual interface? Depends on how it's setup. The easy answer is "yes" - the long answer is we'll have to write a net-scripts module to handle it. If I get around to testing xen, I'll see if a module needs writing, otherwise I'm just passing some ideas around. You could always post up the relevant xen network scripts so I can see what they're doing and what's involved with writing a module.
It's easier for you to emerge xen ;-) Seriously, it won't harm anything on your system, and if you cut and paste the wiki commands and use the default kernel configs you should have it up and running pretty quickly. Even without it running you will be able to get a reasonable idea of what's going on from the scripts in /etc/xen.
This issue has been discussed on xen-users ML. http://lists.xensource.com/archives/html/xen-users/2005-11/msg00038.html http://lists.xensource.com/archives/html/xen-users/2005-11/msg00065.html
i solved all issues with network-bridge script just using RC_NET_STRICT_CHECKING="lo" in /etc/conf.d/rc as was posted in xen-users http://lists.xensource.com/archives/html/xen-users/2005-11/msg00039.html
*** Bug 114794 has been marked as a duplicate of this bug. ***
http://gentoo-wiki.com/Talk:HOWTO_Xen_and_Gentoo#Rollback_network_bridge has some notes on using the gentoo bridging scripts for xen.
There are obviously a few issues with integrating the Xen network scripts and the gentoo ones. Can someone please post a patch if they have xen working reliably with gentoo style bridge setup? (I had a system with udev rules for Xen vif* devices triggering scripts to bring up/down a vlan bridge for each domU, so it is possible to get your stuff set up reliably with xen, but it wasn't using the gentoo networking scripts).
What I did to make networking in XEN work (my interface is eth1): cd /etc/init.d ln -s net.lo net.br0 Add the following in /etc/conf.d/net: ##BEGIN## bridge_br0="eth1" config_eth1=( "null" ) config_br0=( "dhcp" ) depend_br0() { need net.eth1 } brctl_br0=( "setfd 0" "sethello 0" "stp off" ) ##END## reboot the machine Now my network does not work by default so I run: /etc/init.d/net.br0 start Now everyting works like a charm, networking in all domains.
(In reply to comment #7) > > veth0/vif0 = virtual interface? Depends on how it's setup. > The easy answer is "yes" - the long answer is we'll have to write a net-scripts > module to handle it. > > If I get around to testing xen, I'll see if a module needs writing, otherwise > I'm just passing some ideas around. You could always post up the relevant xen > network scripts so I can see what they're doing and what's involved with writing > a module. There are a wiki [1] that can help to understand xen networking. It would be help us (xen users) a lot if we can have this problem. At the moment, any services that "need net" have to start after xend or they will be silently went back to the "stop" state. If there are any more info that you need to have gentoo network script working nicely with xend, please ask or ping me.
oops, I forgot the link. http://wiki.xensource.com/xenwiki/XenNetworking
Right chaps - I finally have a working xen install + networking :) 1) Don't use the xen bridge as that will causes issues described here if you restart xend 2) Create your own bridge (br0) as descrbibed here http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=4&chap=3#doc_chap7 and add your physical interfaces too it. 3) Add this bridge to your /etc/xen/ttylinux or other config file vif = [ 'bridge=br0' ] You now have bridge support working in xen :) Caveats 1) ensure that bridge names have a number at the end otherwise you may encounter a bug in all gentoo baselayouts that will be fixed in 1.12.0_pre20 2) We don't rename any interfaces - you may get confused with xen interface names as a result. 3) We've added the physical interface to the bridge - we should be adding the virtual interface as well but I couldn't get this to work. Hopefully a Xen guy like chrb can comment on this :) OR you guys could just use routing. Anyway, now hopefully that very bad depend can be removed from the xend init script - it should now just need net :)
One final note - udev>=089 users may need to stop hotplug from starting your bridges like so RC_PLUG_SERVICES="!net.br0" I'll see if I can address that if possible.
(In reply to comment #17) > 3) We've added the physical interface to the bridge - we should be adding the > virtual interface as well but I couldn't get this to work. Hopefully a Xen guy > like chrb can comment on this :) > Roy, IIRC you tried to add veth0 to the bridge. The virtual interface should be named peth0. Have you tried that yet? I don't a xen box available ATM to test.
(In reply to comment #19) > Roy, > IIRC you tried to add veth0 to the bridge. There was an error adding it. I haven't investigated why yet. Adding the real interface works like a charm though. > The virtual interface should be named peth0. > Have you tried that yet? I don't a xen box available ATM to test. I see no reason to rename the interface needlessly - I don't know why xen does this. The name doesn't affect adding it to the bridge - I renamed it eth1, peth1 and wibblywoo but same error.
http://wiki.xensource.com/xenwiki/XenNetworking is the place to look. The interface is renamed because "It is good to have the physical interface and the dom0 interface separated; thus you can e.g. setup a firewall on dom0 that does not affect the traffic to the domUs (just for protecting dom0 alone)."
(In reply to comment #21) > http://wiki.xensource.com/xenwiki/XenNetworking is the place to look. OK, you look and you supply documentation for Gentoo Xen setup in relation to bridging and routing. > The interface is renamed because "It is good to have the physical interface and the > dom0 interface separated; thus you can e.g. setup a firewall on dom0 that does > not affect the traffic to the domUs (just for protecting dom0 alone)." > Good for them. We also provide tools to rename interfaces - although our method is a little clumsy it can be done. The fact that an interface is renamed does not alter the interface or number of in any way or form. I could rename eth0 to be wibblywoo and wifi0 to be foobar if I wanted too. That does not affect firewalls, diverting traffic or protecting dom0 - it's just a name.
The real dom0 interface is put into promiscuous mode so it will receive packets for all domains. This usually isn't what people want for dom0 itself, so that interface is renamed, and a new "fake" eth0 interface created that receives the traffic destined for vif0.0, which is one of the virtual interfaces created in dom0 (the virtual interfaces are connected to xen ethernet devices in domains as if by a crossover cable). After that any tool or networking script will be operating on the "fake" device rather than the real one, so running say /etc/init.d/net.eth0 up will bring up the fake device and not the real one (which is now permanently up and has no ip address). It should be possible to not rename the device and just get used to using veth0, but they do so that the switch is transparent to the user and all of the network scripts they already have. Only peth* and vif*.* devices should be attached to the bridge. The xen drivers connect: vif0.0 <--> eth0 (renamed from veth0) vif0.1 <--> eth1 (renamed from veth1) etc.
As I understand it, by making the physical device peth0 and operating the configuration scripts on veth0 (which has been renamed eth0), Xen is in effect creating a virtualized interface for dom0 similar to those it creates for all domU's. Note that xen-br0 includes peth0, vif0.*, and vif*.*. Also, note that the dom0 and domU eth* interfaces are not included in the bridge, but rather connect directly to a corresponding vif*.*. This promotes a consistency of operation, such that the dom0 and domU kernels do not have to differ so much. What results is that all traffic is routed through the vif*.* across the bridge to peth0, from where it goes out into the world. However, I believe the snag lies in the configuration of peth0. My experience was that when Xen networking started, I lost internet access, because while my eth0 had the proper MAC address, it lacked an IP address. Furthermore, the new peth0 interface had the MAC address FE:FF:FF:FF:FF:FF and no IP address. I presume that with this bridging, it is peth0 that should have DHCP run on it or obtain an "real" IP address, as it is the physical interface. Eth0 (on dom0) can then be treated like the domU eth0's and be assigned whatever IP address the user desires. In my mind, this explains the failures of scripts like net.eth0 that I experience, as the script is running on a virtual interface and will receive no IP address since the physical interface (peth0) has no IP address and thus cannot talk to the network. Perhaps the key to integrating Gentoo with Xen bridging lies in this detail, in which case we should probably try modifying one of Xen's scripts to account for this change. I gather that the reason the Xen folk haven't considered the approach I mentioned above is because they expect the eth* of the vif's to largely speak for themselves. Peth0 then acts more like a gateway than anything else, and packets simply flow through it. So, by this logic I should expect my dom0's virtualized eth0 to obtain its own IP address without a hitch, as peth0 should not interfere. However, as I indicated earlier this seems to be broken for me. Perhaps the key lies in a second suggestion: instead of operating the networking scripts on peth0, I should create iptables rules to forward all packets to eth0. However, I have a couple questions on this approach: I assume that packets flowing from domU's carry the IP addresses assigned to the domU eth*'s. If we assume that these packets are unmodified, then they should pass across the bridge and through peth0 onto the internet without a change in IP address. However, if my ttylinux domU for example has been assigned the address 10.10.10.10, I assume that there exists some other client on the internet with the same address. This assumption, combined with the other that the IP addresses on the packets are unmodifed by either the bridge or peth0 would cause some confusion in routing a response back to my domU, no? I wonder, then, if instead the bridge should have an IP address of its own and act as a router for each of the vif's. This seems to be what Roy's suggestion tries to do. It bypass vif0.* altogether, and simply create a new bridge with the physical interfaces named as-is. If one follows the Gentoo document that he referenced, there is an option in /etc/conf.d/net to give the bridge an IP address either statically or by DHCP. (Per the networks I work with, I believe I shall have to use DHCP) Thus, the Gentoo dom0 remains as it would under normal (i.e. non-Xen Linux) conditions, and the vif->eth setup is created only for the domU's. The trade-off is that the dom0 is now a router, whereas with the Xen setup the packets could bypass dom0 and you could effectively hide it. I suppose it really comes down to how much security/protection you want for your dom0. -Ricardo-
(In reply to comment #21) > http://wiki.xensource.com/xenwiki/XenNetworking is the place to look. The > interface is renamed because "It is good to have the physical interface and the > dom0 interface separated; thus you can e.g. setup a firewall on dom0 that does > not affect the traffic to the domUs (just for protecting dom0 alone)." I don't catch the point of that! I setup xenbr0 the normal way. xenbr0 bridges eth0 - and xend adds the interfaces of the domUs to xenbr0. My /etc/conf.d/net looks like this: bridge_xenbr0="eth0" config_xenbr0=( "<ip>/<netmask>" ) routes_xenbr0=( "default via <gateway>" ) brctl_xenbr0=( "setfd 0" "sethello 0" "stp off" ) The IP is assigned to xenbr0. My firewall script also filters the traffic of xenbr0, and it doesn't matter at all for the traffic to the domUs. All the things, that /etc/xen/scripts/network-bridge does by default (transferring the config of eth0 to the bridge interface etc.) are so intrusive, that i don't see why gentoo should tolerate them. Instead, gentoo should include baselayout support for setting up the bridge "the xen-way" if that has any advantage over my config. IMHO, the /etc/xen/scripts/network-bridge script should not change the system setup without using gentoo-scripts.
with 3.0.3. I let xen handles the bridge and every thing are fine. nothing fancy in my /etc/conf.d/net . Restart xend doesn't upset other services anymore. The difference between 3.0.2 and 3.0.3 is cleanup the cruft left over that causes problem. xen-3.0.2 has a better if{up/down} functions for those of us doesn't have them but they didn't remove the broken workaround for gentoo. $ cat /etc/conf.d/net config_eth0=( "192.168.1.2/24" ) routes_eth0=( "default via 192.168.1.1" )
If it's of any help, I managed to get peace of mind in two steps: 1) rename interfaces with udev. I use a etc/udev/10-rename-eth.rules like this: KERNEL="eth*", SYSFS{address}="my:ma:ca:dd:re:ss", NAME="peth0" KERNEL="veth0", NAME="eth0" 2) set up xen-style bridging with a etc/init.d/xenbridge script started in 'boot' runlevel. It depends on 'before net' and uses 'ip' and 'brctl' to: shutdown peth0, change its mac to fe:ff:ff:ff:ff:ff, set my 'correct' mac address on eth0 (which was veth0 renamed by udev), create xenbr0 and add peth0 & vif0.0 to it, bring up interfaces and bridge ...then I just start xend in 'default' runlevel, and use etc/conf.d/net to set up eth0 in the 'standard' way. Actually I have two eth and do all that stuff twice, so I end up with xenbr0 and xenbr1. domU's get access to both and everything work really well. Advantages: - works perfectly, and every domain (0 and Us) get access to both LANs on both my ethernet interfaces. - Does not break if I update baselayout, udev, xen and/or xen-tools even if overwriting old config files with 'fresh' new ones. - seems to handle without problems starting and stopping init.d/xend - but I didn't do extensive test on this. - system works without problems if you don't [want to?] start xend for any reason. The bridge will be setup and working anyway, so you end up always using the veth0-renamed-to-eth0 interface through xenbr0 - if you write iptables rules depending on the bridge network topology they'll work even if you don't start xend. Disadvantages: - works only if using Xen and bridging... switching to a 'standard' linux kernel will probably result in non-working net, because it will end up with peth0 and init scripts will try to assign IPs to a non existent eth0. Using routed networking should be a non-issue, one would just not use this scripts and be ok. - needs hard-coded mac addresses - this could probably be solved with some RTFM'ing of udev rules (renaming ethX to pethX without looking at the mac address?) and making the xenbridge init script a little less stupid - needs hard-coded number of ethernet interfaces. Again, it would be really simple to solve in the init script, don't know about udev rules ("rename every ethX to pethX" ?) Putting it all together, if it wasn't for the "hardcoded mac & interfaces" issue, this could be a really 'transparent' drop-in for switching from vanilla linux to a dom0 running in Xen with bridged networking - even before starting xend and setting up other domains. I'm using xen-3.0.2 and xen-tools-3.0.2-r4 on a fresh install of gentoo-hardened/x86_64. /me too sorry for my english.
Created attachment 126593 [details] init.d script to set up bridged virtual network I just refreshed my dom0 installs. xen-tools-3.0.4_p1 still doesn't set up bridged networking correctly on gentoo (fresh install of hardened amd64). So I made the atteched script, just run it with all ethernets down (actually it depends on 'before net', just add it to default softlevel) and it will set up the virtual network for you, with bridges et all. It basically is an evolution from my previous post, this time without hardwiring mac addresses or using udev. Things to note: - you don't need to change xend config. it will Just Work (tm) with your shiny virtual network setup by the script. - the script is safe to run on non-xen kernels, it won't find needed interfaces and won't touch anything - the script is safe to run on xen kernels regardless of xend being started or not, either way you end up with working ethernet interfaces in dom0 - it's possible to observe a lag of some seconds before the bridges get 'up to speed', but it works. this has nothing to do with the script, it's a bridge issue - if you run a setup with this script and xend being started in default, and disable this script without disabling xend, you'll end up with a box with non working network interfaces on the next reboot. think carefully if you don't have physical access possible evolutions of the script: - a better name :) - write the stop function - we already use sed, so we could drop cut - let the user configure which ethX to ignore /or/ - make the script work on one interface at a time, use it with symlinks like net.* (init.d/virtualnet.eth0 virtualnet.eth1 ...) This script is written with bridged networking in mind. I don't use any other form of networking with Xen. In the bridged case, it would be correct to say that xend depends on this one (starting xend without this, on a plain gentoo system, will result in non-working net). But it would be wrong if you don't use bridged net. I didn't investigate if dependencies could be generated onthefly from conf.d data, if it is the case it would be good to make xend depend on this script if you conf.d it for bridged networking. hope this helps. I think gentoo is the best distro for a dom0 and we have a great xen support with portage. it's just a pity we can't correctly set up ethernet!
I think we need to find a better solution. This is too confusing. the xen people are too intrusive with their scripts... and this veth0 is just madness... I have to rewrite my iptables rules completely. Most people just need to be able to have host-only, nat or bridge configuration... net-brige or net-route xen scripts are completely dumb. and now after starting your script: kali init.d # /etc/init.d/xend start * Starting eth0 * Bringing up eth0 * 192.168.150.1/24 * network interface eth0 does not exist * Please verify hardware or kernel module (driver) [ !! ] * ERROR: cannot start xend as net.eth0 could not start this is just ridiculous... It would have been easier to configure vifx that are created when you start a domU but these are not predictable and you can't set it to a known value. For a host only configuration I would prefer to use: brctl_br0=( "setfd 0" "sethello 0" "stp off" ) bridge_br0="dummy0" config_dummy0=( "null") config_br0=( "192.168.10.1/24" ) depend_br0() { need net.dummy0 } and vif = [ 'bridge=br0' ] to my domU config like it was suggested earlier... I can ping the domU from dom0 and vice versa... but I have to figure out what is wrong with the routing. We really need to wake up and get this xen port fixed; the vmware-server is also broken most of the time. Like you said, gentoo would be the perfect distrib for a dom0... what a shame. :(
Actually I don't even need the dummy0 brctl_br0=( "setfd 0" "sethello 0" "stp off" ) bridge_br0="" config_br0=( "192.168.10.1/24" ) is enough to get a "host-only" setup.
I've been using Xen in full production for months now, using an even more complicated network setup, with no funny hacks, or extra init scripts. It's pure gentoo simplicity, and mostly covered by Kim Pedersen and Roy Marples. === /etc/conf.d/net: config_eth0=( "null" ) config_eth4=( "null" ) RC_NEED_bond1="net.eth0 net.eth4" slaves_bond1="eth0 eth4" config_bond1=( "null" ) RC_NEED_br1="net.bond1" bridge_br1="bond1" config_br1=( "192.168.131.111/24" ) brctl_br1=( "setfd 0" "sethello 0" "stp off" ) config_eth1=( "null" ) config_eth5=( "null" ) RC_NEED_bond2="net.eth1 net.eth5" slaves_bond2="eth1 eth5" config_bond2=( "null" ) RC_NEED_br2="net.bond2" bridge_br2="bond2" config_br2=( "192.168.136.111/24" ) brctl_br2=( "setfd 0" "sethello 0" "stp off" ) routes_br2=( "default via 192.168.136.1" ) config_eth3=( "null" ) config_eth7=( "null" ) RC_NEED_bond3="net.eth3 net.eth7" slaves_bond3="eth3 eth7" config_bond3=( "null" ) RC_NEED_br3="net.bond3" bridge_br3="bond3" config_br3=( "192.168.128.111/24" ) brctl_br3=( "setfd 0" "sethello 0" "stp off" ) config_eth2=( "null" ) config_eth6=( "null" ) RC_NEED_bond4="net.eth2 net.eth6" slaves_bond4="eth2 eth6" config_bond4=( "null" ) RC_NEED_br4="net.bond4" bridge_br4="bond4" config_br4=( "192.168.137.111/24" ) brctl_br4=( "setfd 0" "sethello 0" "stp off" ) === /etc/xen/auto/dns1.xen.cfg # $Id: dns1.xen.cfg 29151 2007-08-05 20:33:32Z mikew $ kernel = "/boot/kernel-genkernel-x86_64-xen-2.6.18-xen"; memory = 256; maxmem = 512; name = "dns1"; vif = [ "bridge=br2", "bridge=br4" ]; disk = ['phy:mapper/vgxen-dns1,xvda1,w', 'phy:mapper/vgxen-dns1_swap,xvda2,w']; root = "/dev/xvda1 ro"; vcpus = 1; ==== That server, and it's sisters, have 8 NICs each, bonded in pairs to form 4 usable interfaces, with each bond added to a bridge. The domU config then references the bridge of the bond they are to exist on, in this case the dns server is shared between 2 networks. I have symlinks to net.lo for each and every interface referenced, but only net.br? are added to any runlevel, the RC_NEEDS_... sort the dependencies out. No other configuration is necessary, it Just Works (tm). The domU itself, in the above case, sees eth0 and eth1, those should be setup just like any real machine, complete with default gateway. If you don't have ethernet bonds like I, the net config would be more like: config_eth2=( "null" ) RC_NEED_br2="net.eth2" bridge_br2="eth2" config_br2=( "192.168.136.111/24" ) brctl_br2=( "setfd 0" "sethello 0" "stp off" ) routes_br2=( "default via 192.168.136.1" )
> Actually I don't even need the dummy0 > > brctl_br0=( "setfd 0" "sethello 0" "stp off" ) > bridge_br0="" > config_br0=( "192.168.10.1/24" ) > > is enough to get a "host-only" setup. It never worked for me that way because br0's MAC-address depends on the added interfaces. And the interfaces added by Xen all have MAC-addresses like "fe:ff:ff:ff:ff:ff". And with that MAC-adress, it simply doesn't work. So i'm always adding a tun-interface to have a constant and sane MAC-address.
(In reply to comment #32) > > Actually I don't even need the dummy0 > > > > brctl_br0=( "setfd 0" "sethello 0" "stp off" ) > > bridge_br0="" > > config_br0=( "192.168.10.1/24" ) > > > > is enough to get a "host-only" setup. > > It never worked for me that way because br0's MAC-address depends on the added > interfaces. And the interfaces added by Xen all have MAC-addresses like > "fe:ff:ff:ff:ff:ff". And with that MAC-adress, it simply doesn't work. So i'm > always adding a tun-interface to have a constant and sane MAC-address. > you're right! Back to use the dummy0 version at least I get a mac address. config_dummy0=( "null") config_br0=( "192.168.10.1/24" ) depend_br0() { need net.dummy0 }
anyone using connection tracking? short story is: xen-sources-2.6.16.52 on hardened/amd64, and connection tracking does not work for tcp (I don't get any NEW/ESTABLISHED/RELATED state match), while it seems to work ok for udp and icmp. I found out [https://lists.netfilter.org/pipermail/netfilter/2005-June/061101.html] and, while it shouldn't be related to my kernel, I tried turning off TCP SACK (echo 0 > /proc/sys/net/tcp_sack) and nothing changes. I also tried 'iptables -t raw -A PREROUTING -i bridge1 -j NOTRACK' and still nothing changes. I don't know if this can be my fault in the Xen network bridge config (I'm using the script of my last comment) or if it is a bug in netfilter and it should be filed somewhere else... Using the script of my last comment this is the situation I got [there isn't any other rule other than those shown in the INPUT chain]: # uname -a; lsmod Linux srv1 2.6.16.52-xen0 #2 SMP Sat Aug 4 14:14:30 CEST 2007 x86_64 Intel(R) Xeon(TM) CPU 3.00GHz GenuineIntel GNU/Linux Module Size Used by xt_state 2688 9 ip_conntrack 46876 1 xt_state iptable_filter 3456 1 ip_tables 12152 1 iptable_filter x_tables 11912 2 xt_state,ip_tables # brctl show bridge name bridge id STP enabled interfaces bridge0 8000.feffffffffff no vif0.0 peth0 bridge1 8000.feffffffffff no vif0.1 peth1 # ip addr show eth1 5: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue link/ether 00:14:22:XX:YY:ZZ brd ff:ff:ff:ff:ff:ff inet AA.BB.CC.DD/24 brd AA.BB.CC.255 scope global eth1 inet6 fe80::214:22ff:feXX:YYZZ/64 scope link valid_lft forever preferred_lft forever # iptables -Z; wget -q http://www.kernel.org; iptables -L INPUT -n -v Chain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state ESTABLISHED 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW 21 25983 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 1 110 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 state ESTABLISHED 0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED 1 60 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW 7 2108 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 state ESTABLISHED 0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED 0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW 1 140 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
(In reply to comment #34) > short story is: xen-sources-2.6.16.52 on hardened/amd64, and connection > tracking does not work for tcp (I don't get any NEW/ESTABLISHED/RELATED state > match), while it seems to work ok for udp and icmp. [....] I switched to xen-sources-2.6.20-r2 and the problem disappeared. I had TCP SYNCOOKIES support compiled in 2.6.16.52 but did not enable it at runtime, while I disabled support at compile time in 2.6.20-r2.
(In reply to comment #35) > (In reply to comment #34) > > short story is: xen-sources-2.6.16.52 on hardened/amd64, and connection > > tracking does not work for tcp (I don't get any NEW/ESTABLISHED/RELATED state > > match), while it seems to work ok for udp and icmp. > [....] > > I switched to xen-sources-2.6.20-r2 and the problem disappeared. I had TCP > SYNCOOKIES support compiled in 2.6.16.52 but did not enable it at runtime, > while I disabled support at compile time in 2.6.20-r2. > One problem solved at least :-) This bug has quite a lot of stuff in it, is there anyone still having problems with bridged networking using xen's normal scripts on 3.1.0 or 3.0.4?
(In reply to comment #36) > This bug has quite a lot of stuff in it, is there anyone still having problems > with bridged networking using xen's normal scripts on 3.1.0 or 3.0.4? The scripts still create xenbr0, move eth0's config to xenbr0, and stuff like that, isn't it? Well, that's terrible. For example "net.eth0 restart" will still do moficiations to eth0 instead of xenbr0, and so on ...
(In reply to comment #37) > (In reply to comment #36) > > This bug has quite a lot of stuff in it, is there anyone still having problems > > with bridged networking using xen's normal scripts on 3.1.0 or 3.0.4? > > The scripts still create xenbr0, move eth0's config to xenbr0, and stuff like > that, isn't it? > > Well, that's terrible. For example "net.eth0 restart" will still do > moficiations to eth0 instead of xenbr0, and so on ... > It's all kinda crazy (go go xen for making things crazy complicated) but does work for me. On boot this is what is supposed to happen: * the system starts off with eth0, veth0, and vif0.0 interfaces * eth0 is started with net.eth0 * xend starts ** scripts change eth0 to peth0 ** change veth0 to eth0 ** copy the config from the old eth0 to the new eth0 ** create xenbr0, bridge peth0 and vif0.0 restarting net.eth0 will kick this new eth0, applying the network settings to it as expected and won't harm the bridge at all.
I think I have found another very interesting thing why Bridging kills some people's interfaces. In my case I have ipv6 on my interface too. The network script of Xen expect that there is only ipv4 on the interface and places a 'grep' wrong in netwrok-bridge. Is a space is added to get_ip_info: addr_pfx=`ip addr show dev $1 | egrep '^ *inet ' | sed -e 's/ *inet //' -e "s/$1//"` ('^ *inet ' instead of '^ *inet') Then the script results the right config and the code that uses addr_pfx suddenly works. In anycase, of course it would be great if it compensated for IPv6 too.
*** Bug 209240 has been marked as a duplicate of this bug. ***
(In reply to comment #39) > I think I have found another very interesting thing why Bridging kills some > people's interfaces. In my case I have ipv6 on my interface too. The network > script of Xen expect that there is only ipv4 on the interface and places a > 'grep' wrong in netwrok-bridge. Oh yes. I hate these lines. > Is a space is added to get_ip_info: > > addr_pfx=`ip addr show dev $1 | egrep '^ *inet ' | sed -e 's/ *inet //' -e > "s/$1//"` > > ('^ *inet ' instead of '^ *inet') > > Then the script results the right config and the code that uses addr_pfx > suddenly works. In anycase, of course it would be great if it compensated for > IPv6 too. If you have multiple IPs (IPv4 and/or IPv6) on the interface the grep returns more than one line and "ip addr add ${addr_pfx} dev $1" will fail. This workaround works only if the grep returns now only one line. That doesn't fix the whole problem. I found another approrach to fix this killing interface behavior based on http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=332. --- network-bridge.dist 2008-02-13 01:18:07.000000000 +0100 +++ network-bridge 2008-02-13 04:33:40.000000000 +0100 @@ -90,7 +90,7 @@ tdev=tmpbridge get_ip_info() { - addr_pfx=`ip addr show dev $1 | egrep '^ *inet' | sed -e 's/ *inet //' -e "s/$1//"` + addr_pfx=`ip addr show dev $1 | grep -v dynamic | egrep '^ *inet' | sed -e 's/ *inet6\? //' -e "s/$1//"` gateway=`ip route show dev $1 | fgrep default | sed 's/default via //'` } @@ -99,7 +99,7 @@ if [ -n "$addr_pfx" ] ; then # use the info from get_ip_info() ip addr flush $1 - ip addr add ${addr_pfx} dev $1 + for address in `echo "${addr_pfx}"`; do ip addr add ${address} dev $1; done ip link set dev $1 up [ -n "$gateway" ] && ip route add default via ${gateway} fi This patch add support for multiple IPv4 and v6 addresses on the interface.
(In reply to comment #31) > I've been using Xen in full production for months now, using an even more > complicated network setup, with no funny hacks, or extra init scripts. It's > pure gentoo simplicity, and mostly covered by Kim Pedersen and Roy Marples. > === > > /etc/xen/auto/dns1.xen.cfg > # $Id: dns1.xen.cfg 29151 2007-08-05 20:33:32Z mikew $ > > kernel = "/boot/kernel-genkernel-x86_64-xen-2.6.18-xen"; > memory = 256; > maxmem = 512; > name = "dns1"; > vif = [ "bridge=br2", "bridge=br4" ]; > disk = ['phy:mapper/vgxen-dns1,xvda1,w', 'phy:mapper/vgxen-dns1_swap,xvda2,w']; > root = "/dev/xvda1 ro"; > vcpus = 1; > > ==== > config_eth2=( "null" ) > RC_NEED_br2="net.eth2" > bridge_br2="eth2" > config_br2=( "192.168.136.111/24" ) > brctl_br2=( "setfd 0" "sethello 0" "stp off" ) > routes_br2=( "default via 192.168.136.1" ) > We've still been using this exact setup with no other configuration changes necessary for something like a year now. However, today we found a problem with Xen 3.2. Xen have decided to drop xenbr${vif} in favour of making bridges called eth${vif}. xend now seems to ignore the fact we've already setup a bridge and screws up the networking by doing it's eth -> peth thing. Fixing this is simply a matter of commenting out "(network-script network-bridge)" from /etc/xen/xend-config.sxp. Xen now does not attempt to setup bridging, letting us continue to do it properly. Mike
(In reply to comment #42) > We've still been using this exact setup with no other configuration changes > necessary for something like a year now. > However, today we found a problem with Xen 3.2. Xen have decided to drop > xenbr${vif} in favour of making bridges called eth${vif}. xend now seems to > ignore the fact we've already setup a bridge and screws up the networking by > doing it's eth -> peth thing. > Fixing this is simply a matter of commenting out "(network-script > network-bridge)" from /etc/xen/xend-config.sxp. > Xen now does not attempt to setup bridging, letting us continue to do it > properly. > > Mike > That is the correct thing to do, if you want to manage bridges manually or require a more complicated setup simply disable the network-bridge script or replace it. The point of the network-bridge script is just to try to make simple setups just work. On a side note ipv6 is still broken but I do want to get that fixed at some point.
(In reply to comment #43) > (In reply to comment #42) > > We've still been using this exact setup with no other configuration changes > > necessary for something like a year now. > > However, today we found a problem with Xen 3.2. Xen have decided to drop > > xenbr${vif} in favour of making bridges called eth${vif}. xend now seems to > > ignore the fact we've already setup a bridge and screws up the networking by > > doing it's eth -> peth thing. > > Fixing this is simply a matter of commenting out "(network-script > > network-bridge)" from /etc/xen/xend-config.sxp. > > Xen now does not attempt to setup bridging, letting us continue to do it > > properly. > > > > Mike > > > > That is the correct thing to do, if you want to manage bridges manually or > require a more complicated setup simply disable the network-bridge script or > replace it. The point of the network-bridge script is just to try to make > simple setups just work. > > On a side note ipv6 is still broken but I do want to get that fixed at some > point. Someone at ##xen@irc.freenode.net was also 'fixing' a lot of Xen scripts.
(In reply to comment #43) > That is the correct thing to do, if you want to manage bridges manually or > require a more complicated setup simply disable the network-bridge script or > replace it. The point of the network-bridge script is just to try to make > simple setups just work. Sorry, people keep posting crazy complicated answers, when it's really so simple :) Just stop xen trying to setup things up, and let gentoo do it for you. > On a side note ipv6 is still broken but I do want to get that fixed at some > point. Broken? Odd. dannii ~ # xm list Name ID Mem VCPUs State Time(s) Domain-0 0 192 2 r----- 131135.0 lvs 9 512 1 -b---- 22084.0 openvpn 6 96 1 -b---- 7173.7 webproxy-v6 7 96 1 -b---- 24584.2 dannii ~ # ping6 www.kame.net PING www.kame.net(orange.kame.net) 56 data bytes 64 bytes from orange.kame.net: icmp_seq=1 ttl=45 time=387 ms 64 bytes from orange.kame.net: icmp_seq=2 ttl=45 time=419 ms --- www.kame.net ping statistics --- 3 packets transmitted, 2 received, 33% packet loss, time 1998ms rtt min/avg/max/mdev = 387.642/403.526/419.410/15.884 ms webproxy-v6 ~ # ping6 www.kame.net PING www.kame.net(orange.kame.net) 56 data bytes 64 bytes from orange.kame.net: icmp_seq=1 ttl=45 time=436 ms 64 bytes from orange.kame.net: icmp_seq=2 ttl=45 time=436 ms 64 bytes from orange.kame.net: icmp_seq=3 ttl=45 time=434 ms 64 bytes from orange.kame.net: icmp_seq=4 ttl=45 time=456 ms 64 bytes from orange.kame.net: icmp_seq=5 ttl=45 time=478 ms --- www.kame.net ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 3998ms rtt min/avg/max/mdev = 434.544/448.532/478.774/17.199 ms Seems to work here just fine :) I can't say for sure that I've not changed anything, but I really really don't think I have done anything to that machine. It's not doing anything important so I can poke around at it if it'd help. Mike
(In reply to comment #41) > --- network-bridge.dist 2008-02-13 01:18:07.000000000 +0100 > +++ network-bridge 2008-02-13 04:33:40.000000000 +0100 > @@ -90,7 +90,7 @@ > tdev=tmpbridge > > get_ip_info() { > - addr_pfx=`ip addr show dev $1 | egrep '^ *inet' | sed -e 's/ *inet //' -e > "s/$1//"` > + addr_pfx=`ip addr show dev $1 | grep -v dynamic | egrep '^ *inet' | sed -e > 's/ *inet6\? //' -e "s/$1//"` > gateway=`ip route show dev $1 | fgrep default | sed 's/default via //'` > } > > @@ -99,7 +99,7 @@ > if [ -n "$addr_pfx" ] ; then > # use the info from get_ip_info() > ip addr flush $1 > - ip addr add ${addr_pfx} dev $1 > + for address in `echo "${addr_pfx}"`; do ip addr add ${address} dev > $1; done > ip link set dev $1 up > [ -n "$gateway" ] && ip route add default via ${gateway} > fi > Doesn't work at my server this way, because "for" takes after the IP/mask the next word - "brd"(broadcast), and then "ip addr add" fails I have added some commands, so that the whole line from addr_pfx is added per "ip addr add" (but only till the word "global", cause "secondary" etc. for eth0:1 or so is also not excepted through ip addr add): - for address in `echo "${addr_pfx}"`; do ip addr add ${address} dev $1; done + addr_z=`echo -e "${addr_pfx}" | wc -l` for ((i=1;i<=$addr_z;i++)); do address=`echo -e "${addr_pfx}" | sed -n ${i}p | grep -o ^.*global` ip addr add ${address} dev $1; done after that I have eth0 und eth0:1, and peth0 - I don't know if there should be something else, i am new to Xen and Gentoo
lalala
(In reply to comment #47) > lalala > o_O ?
I just built a new system with xen-3.2.1 from portage. It has two ethernet interfaces and I assign multiple IPv4s to them in the standard gentoo way, they also get the default IPv6 addresses (IPv6 enabled kernel). The out-of-the-box xen network-bridge scripts will kill any network access to the machine. The patch of comment #41 won't do the trick. A setup like comment #42 works like a charm and we're using that.
Created attachment 165963 [details, diff] xen-tools-3.3.0-network-bridge-multiple-ips.patch (In reply to comment #46) > Doesn't work at my server this way, because "for" takes after the IP/mask the > next word - "brd"(broadcast), and then "ip addr add" fails Mh, yes. You're right. While setting up a new machine a ran into the same problems as last time. So I reworked the patch. See the attachment. Tested so far: * multiple IPv4 ips * multiple static IPv6 ips * starting and stopping from command line As working on the "op_stop" command I discovered that Xend isn't stopping the network at any time. Only the "start" command gets called. Two times on /etc/init.d/xend start and one time on /etc/init.d xend (status|stop). In comments on xen-devel@lists.xensource.com it's called a 'feature'. (http://article.gmane.org/gmane.comp.emulators.xen.devel/15659) Whatever... With the patch it should be possible to use network-bridge as network-script to create a bridge at startup. To revert all changes you have to call /etc/xen/scripts/network-bridge stop manually (e.g. in the init.d script) An other way is to discard xens network-scripts and creating the bridge on your own using /etc/conf.d/net like suggested earlier. You choose.
Hello friends! I've found another exciting issue with the network-bridge script. Static host routes do not seem to be transferred successfully, and if the gateway requires the static host route then adding it fails as well. I don't have time to debug this issue and have been convinced by it to switch my hosts to the more robust conf.d/net configuration, but I figured I'd make this information available for anyone looking at cleaning up network-bridge.
Xen 4.1 in tree. Please test with it and reopen if it doesnt work
