Created attachment 305813 [details] /var/log/rc.log Syslog-ng is not started after a reboot. I usen an external logserver for storing a copy of the syslog and hence syslog-ng requires that the network is up and configured. My situation is this: I have 3 network interfaces: eth0 to the internal LAN, eth1 to the Internet, and eth2 to my wireless LAN. All three interfaces are in the 'default' runlevel. In spite of these lines in the syslog-ng start script: depend() { # Make networking dependency conditional on configuration case $(sed 's/#.*//' /etc/syslog-ng/syslog-ng.conf) in *source*tcp*|*source*udp*|*destination*tcp*|*destination*udp*) need net.eth0 after net.eth0 use stunnel ;; esac config /etc/syslog-ng/syslog-ng.conf use clock need hostname localmount provide logger } (where I have changed 'need net' to 'need net.eth0' and added 'after net.eth0') syslog-ng is attempted started *before* eth0. I have uncommented this line in /etc/openrc.conf: rc_depend_strict="YES" Which should - at least as I read the comment - ensure that all network interfaces should be strated before 'net' is made available. The log from OpenRC is attached. Also attached is the relevant (I hope) portion of /lib/rc/init.d/deptree
Created attachment 305815 [details] syslog-ng part of /lib/rc/init.d/deptree
(In reply to comment #0) > My situation is this: I have 3 network interfaces: eth0 to the internal LAN, > eth1 to the Internet, and eth2 to my wireless LAN. All three interfaces are > in the 'default' runlevel. In spite of these lines in the syslog-ng start > script: > > depend() { > # Make networking dependency conditional on configuration > case $(sed 's/#.*//' /etc/syslog-ng/syslog-ng.conf) in > > *source*tcp*|*source*udp*|*destination*tcp*|*destination*udp*) > need net.eth0 > after net.eth0 > use stunnel ;; > esac > > config /etc/syslog-ng/syslog-ng.conf > use clock > need hostname localmount > provide logger > } > (where I have changed 'need net' to 'need net.eth0' and added 'after > net.eth0') Please do not modify the startup script; put that back to the way it was. > > > rc_depend_strict="YES" You don't need this either. Put the following line in /etc/conf.d/syslog-ng: rc_need="!net net.eth0" That is the only change you should need. Report back to the bug whether that makes it start correctly.
Created attachment 305829 [details] rc.log - version2
Created attachment 305831 [details] syslog-ng part of /lib/rc/init.d/deptree versio2
I have made the suggested changes and rebooted the server, but the situation is the same: syslog-ng is not started. I have attached updated versions of rc.log and the syslog-ng part of deptree.
Looking at the rc.log you attached, it looks like there is an issue with your configuration for net.eth0. Do you mind attaching /etc/conf.d/net?
Created attachment 306559 [details] /etc/conf.d/net
(In reply to comment #6) > Do you mind attaching /etc/conf.d/net? Done. Please note that the attached version is the old, original net from before OpenRC was introduced updated as described in the migration guide (http://www.gentoo.org/doc/en/openrc-migration.xml). One additional note: If I move net.eth0 to the 'boot' runlevel it starts of course before syslog-ng, but - and this is the interesting bit - the 2nd and 3rd line of config_eth0="192.168.19.6 netmask 255.255.255.0 broadcast 192.168.19.255 2001:470:91c2:19::6/64 fe80::19/64" are not executed. When it is run in runlevel 'default' they are. Any clue?
(In reply to comment #8) > (In reply to comment #6) > > > Do you mind attaching /etc/conf.d/net? > > Done. Please note that the attached version is the old, original net from > before OpenRC was introduced updated as described in the migration guide > (http://www.gentoo.org/doc/en/openrc-migration.xml). So this is not the net file you are using? If it isn't, please attach the one you are using. Also, net.eth0 should be in the default runlevel not the boot runlevel. Also, do the following: /etc/init.d/net.eth0 stop /etc/init.d/net.eth0 --debug start >& eth0.log and attach that file. Thanks, William
(In reply to comment #9) > > So this is not the net file you are using? > It is the one I use. I expressed myself clumsily. Sorry. > Also, net.eth0 should be in the default runlevel not the boot runlevel. It is. I just tried it as one of my many experiments. > Also, do the following: > > /etc/init.d/net.eth0 stop > /etc/init.d/net.eth0 --debug start >& eth0.log > Easier said than done. I normally access the (headless) server through eth0, but using a serial console (my laptop) I succeeded. > and attach that file. One observation: Bringing down eth0 stopped all the services (Asterisk, Apache, Postfix, etc) When I restarted eth0, these services were not brought back up again. Is that normal behavour?
Created attachment 306565 [details] Debug output from bringing eth0 up.
Do you have iproute2 in your system right? Is IPv6 compiled as a module maybe? If so, how early do you load the ipv6 module? From the the log you attached, these lines are present: + ip addr add 2001:470:91c2:19::6/64 dev eth0 + ip addr add fe80::19/64 dev eth0 So it definetly tried to add the IPs, and I don't see any errors. Do: /etc/init.d/net.eth0 --verbose restart ip addr And attach the full output of those together.
Created attachment 306657 [details] Log file for eth0 restart + ip addr list
(In reply to comment #12) > Do you have iproute2 in your system right? Is IPv6 compiled as a module > maybe? If so, how early do you load the ipv6 module? > I believe that my IPv6 setup is right. It has been working well for years. IPv6 is compiled as a module. I do not do anything special in order to get it loaded. My kernel is configures to autolaod required modules. > Do: > /etc/init.d/net.eth0 --verbose restart > ip addr > > And attach the full output of those together. The output from 'ip addr list' is at the end of the attachment.
Ok, so the only case I can see, is if the ipv6 module isn't loaded early enough in boot, then the adding of v6 addresses will fail during that, but later, after the module is loaded, it works fine.
(In reply to comment #15) > Ok, so the only case I can see, is if the ipv6 module isn't loaded early > enough in boot, then the adding of v6 addresses will fail during that, but > later, after the module is loaded, it works fine. I have added these two lines to /etc/conf.d/modules: modules_3="ipv6" modules_3="sit" but the result is still the same: syslog-ng is not started.
Here are some thoughts of perhaps a more filosophical character. I hope this is the right place to do this kind of things ;-) It has to do with the way openRC handles dependencies, specifically the virtual service 'net' Im my situation I have four network interfaces (lo, eth0, eth1, and eth2) that all provide 'net'. I find it a bit odd that when you shut down just one of them, then all services that need 'net' are shut down. Why? There are still 3 network interfaces running, so 'net' is still provided. /etc/rc.conf has the possibilty to require that all services providing e.g. 'net' should be started before the dependency is satistied, but I have been told in comment #2 (https://bugs.gentoo.org/show_bug.cgi?id=408753#c2) that I should not use rc_depend_strict="YES". What, then, is the purpose of this variable? Finally, /etc/init.d/syslog-ng needs 'net' but it still complains over eth0 not being started. How does it it find this out? /Bent
(In reply to comment #16) > I have added these two lines to /etc/conf.d/modules: > > modules_3="ipv6" > modules_3="sit" > > but the result is still the same: syslog-ng is not started. That overrides it to just the value of 'sit'. It would need to be modules_3="sit ipv6" In your boot output, do modules get loaded before net.eth0 is brought up?
(In reply to comment #17) > Here are some thoughts of perhaps a more filosophical character. I hope this > is the right place to do this kind of things ;-) It has to do with the way > openRC handles dependencies, specifically the virtual service 'net' > > Im my situation I have four network interfaces (lo, eth0, eth1, and eth2) > that all provide 'net'. I find it a bit odd that when you shut down just one > of them, then all services that need 'net' are shut down. Why? There are > still 3 network interfaces running, so 'net' is still provided. lo does NOT provide net. Please see the discussion on the gentoo-dev list about how that has changed, and how users with many interfaces should customized their systems to specify what their local definition of 'net' actually is. > Finally, /etc/init.d/syslog-ng needs 'net' but it still complains over eth0 > not being started. How does it it find this out? Change the dependency via the conf.d to explicitly state what interface should be used to bring up syslog-ng.
(In reply to comment #18) > > That overrides it to just the value of 'sit'. > It would need to be > modules_3="sit ipv6" > I made the change and rebooted. No chane. > In your boot output, do modules get loaded before net.eth0 is brought up? Boot output?? dmesg or rc.log? Here is an excerpt from dmesg that to me shows that the modules are loaded early: ........... EXT3-fs (sda1): mounted filesystem with writeback data mode VFS: Mounted root (ext3 filesystem) readonly on device 8:1. Freeing unused kernel memory: 292k freed NET: Registered protocol family 1 <--- ****** IPv4 udevd[922]: starting version 171 .... lines omitted hub 2-0:1.0: 4 ports detected NET: Registered protocol family 10 <--- ****** IPv6 IPv6 over IPv4 tunneling driver EXT3-fs (sda1): using internal journal kjournald starting. Commit interval 5 seconds EXT3-fs (sdb5): using internal journal EXT3-fs (sdb5): mounted filesystem with writeback data mode kjournald starting. Commit interval 5 seconds EXT3-fs (sdb6): using internal journal EXT3-fs (sdb6): mounted filesystem with writeback data mode Adding 987960k swap on /dev/sdb1. Priority:-1 extents:1 across:987960k ip6_tables: (C) 2000-2006 Netfilter Core Team nf_conntrack version 0.5.0 (8062 buckets, 32248 max) ip_tables: (C) 2000-2006 Netfilter Core Team via-rhine 0000:00:06.0: eth0: link up, 100Mbps, full-duplex, lpa 0xCDE1 via-rhine 0000:00:07.0: eth1: link up, 100Mbps, full-duplex, lpa 0xC5E1 eth0: no IPv6 routers present eth0: no IPv6 routers present eth1: no IPv6 routers present via-rhine 0000:00:08.0: eth2: link up, 100Mbps, full-duplex, lpa 0x45E1 eth2: no IPv6 routers present eth2: no IPv6 routers present
(In reply to comment #19) > > lo does NOT provide net. Please see the discussion on the gentoo-dev list > about how that has changed, and how users with many interfaces should > customized their systems to specify what their local definition of 'net' > actually is. > It took me a while to find the posts - Gentoo's list archive does not have a search function. I read the posts and will only suggest that the information should be publicized better - e.g. in the the Gentoo Manual. 'gentoo-dev' is not the first place you search for configuration information. > Change the dependency via the conf.d to explicitly state what interface > should be used to bring up syslog-ng. I have already had the line rc_need="!net net.eth0" in /etc/conf.d/syslog-ng for several days,but it does not provide the desired effect. I can't stop thinking that we are looking at a bug in openRC or one of its components.
WilliamH: - Is there a way we can get the net.* to always give verbose output during their normal startup? - Can we get a printed message for the console & rc.log that says when the runlevel is switched? (In reply to comment #20) > (In reply to comment #18) > > In your boot output, do modules get loaded before net.eth0 is brought up? > Boot output?? dmesg or rc.log? To start with, how about what's visible on the screen? rc.log should have it as well. > Here is an excerpt from dmesg that to me shows that the modules are loaded > early: Deducing from your output here. > ........... > EXT3-fs (sda1): mounted filesystem with writeback data mode > VFS: Mounted root (ext3 filesystem) readonly on device 8:1. > Freeing unused kernel memory: 292k freed > NET: Registered protocol family 1 <--- ****** IPv4 > udevd[922]: starting version 171 sysinit/udev start > > .... lines omitted > > hub 2-0:1.0: 4 ports detected > NET: Registered protocol family 10 <--- ****** IPv6 > IPv6 over IPv4 tunneling driver boot/modules start > EXT3-fs (sda1): using internal journal > kjournald starting. Commit interval 5 seconds root is remounted > EXT3-fs (sdb5): using internal journal > EXT3-fs (sdb5): mounted filesystem with writeback data mode > kjournald starting. Commit interval 5 seconds > EXT3-fs (sdb6): using internal journal > EXT3-fs (sdb6): mounted filesystem with writeback data mode boot/localmount > Adding 987960k swap on /dev/sdb1. Priority:-1 extents:1 across:987960k boot/swap > ip6_tables: (C) 2000-2006 Netfilter Core Team > nf_conntrack version 0.5.0 (8062 buckets, 32248 max) > ip_tables: (C) 2000-2006 Netfilter Core Team ???/iptables > via-rhine 0000:00:06.0: eth0: link up, 100Mbps, full-duplex, lpa 0xCDE1 boot/net.eth0 > via-rhine 0000:00:07.0: eth1: link up, 100Mbps, full-duplex, lpa 0xC5E1 ???/net.eth1 > via-rhine 0000:00:08.0: eth2: link up, 100Mbps, full-duplex, lpa 0x45E1 ???/net.eth2 What (In reply to comment #21) > (In reply to comment #19) > > > > lo does NOT provide net. Please see the discussion on the gentoo-dev list > > about how that has changed, and how users with many interfaces should > > customized their systems to specify what their local definition of 'net' > > actually is. > > > It took me a while to find the posts - Gentoo's list archive does not have a > search function. I read the posts and will only suggest that the information > should be publicized better - e.g. in the the Gentoo Manual. 'gentoo-dev' is > not the first place you search for configuration information. It's a change functionality that needs documentation still. > > Change the dependency via the conf.d to explicitly state what interface > > should be used to bring up syslog-ng. > > I have already had the line > > rc_need="!net net.eth0" > > in /etc/conf.d/syslog-ng for several days,but it does not provide the > desired effect. I can't stop thinking that we are looking at a bug in openRC > or one of its components. I've written before about debugging circular dep loops. Can you please tar up your: /etc/conf.d/ /etc/init.d/ /etc/runlevels/ and email them to me? Replace passwords, but as little else as possible please. I've got a very rough tool that reads in the above and graphs all of the possible deptree solutions, so we can try to spot the circular loops.
Created attachment 307019 [details] Captured output from a reboot of the server in question.
(In reply to comment #22) > > Boot output?? dmesg or rc.log? > To start with, how about what's visible on the screen? > rc.log should have it as well. > I have uploaded the complete capture file from a restart of the server that exibits the error in question. The server is headless so I made the file by using the capture function of minicom running on my laptop connected to the server via a serial line. It is ot much different from rc.log except for positive proof that the ipv6 module is loaded early.
(In reply to comment #22) Robin, I have sent you a couple of e-mails. Did you ever receive them? Kind regards, Bent
Bent: sorry, I was away at a conference. Where is the tarball I asked for in comment 22?
Still no response from user.
I sent you the requested information on March 27 and again - as I got no response - on April 16. As per request I send it as a provate message. So...
(In reply to comment #28) > I sent you the requested information on March 27 and again - as I got no > response - on April 16. As per request I send it as a provate message. So... I don't have an email from you dated March 27th. I have two: April 16th, and May 10th, neither of which have an attachments.
Perhaps the mail system at gentoo.org strips off attachments. Anyway, you can get the file here: http://www.oz6bl.dk/bug408753.bz2 Please give me a note when you have got so I can take it down again.
(In reply to comment #30) > Perhaps the mail system at gentoo.org strips off attachments. Nope, I know they work. But the mail you mention of March 27th is nowhere in our gentoo.org mail logs (I checked). > Anyway, you can get the file here: http://www.oz6bl.dk/bug408753.bz2 > > Please give me a note when you have got so I can take it down again. Got it
williamh: in comment 22, I asked you for a way to get an entire boot to be verbose, like the output from "/etc/init.d/net.eth0 --verbose start". I didn't see any response. bent: Ok, that's proven that there is no circular dep in your setup at all. It's only that net.eth0 is failing to come up the first time - or maybe openrc isn't trying, but I don't think that is likely. The only other thing I can think of is maybe that the rhine card you have takes a few seconds to warm up for use. can you add this to the conf.d/net and reboot to get rc.log output again? preup() { einfo "start preup $IFACE" /usr/sbin/ethtool $IFACE /sbin/ip addr show dev $IFACE einfo "end preup $IFACE" }
Created attachment 312381 [details] rc.log after adding lines to preup()
I have uploaded the rc.log taken after adding your lines to preup. The last few lines in the log showing syslog-ng starting are due to my having put a file 'syslog.start' in /etc/local.d with these lines: #!/bin/bash # ebegin "(re)Starting syslog-ng" # /etc/init.d/syslog-ng restart eend "done"
Bent: Thanks. So it really does look like it never even tried to run net.eth0, and I'm even more confused now :-(. Can you tar up /lib/rc/init.d/ and attach it? (There is nothing private/secret in there). WilliamH: Can you think of why it thinks that net.eth0 cannot start?
Sorry I didn't see this comment before. (In reply to comment #22) > WilliamH: > - Is there a way we can get the net.* to always give verbose output during > their normal startup? Try putting VERBOSE_OUTPUT=true in /etc/conf.d/net. > - Can we get a printed message for the console & rc.log that says when the > runlevel is switched? I'm not sure what you mean here, because rc.log always shows when you switch to a new runlevel. (In reply to comment #35) > WilliamH: > Can you think of why it thinks that net.eth0 cannot start? If you look at the log, net.eth0 does start. It just starts after we attempt to start syslog-ng for some reason.
Created attachment 312393 [details] /lib/rc/init.d/*
bent: I'm grasping at straws here now, but let's try a few bits. in conf.d/iptables AND conf.d/ip6tables, please add: rc_before="!net" rc_use="!logger"
Created attachment 312415 [details] rc.log after adding lines to conf.d/iptables and conf.d/ip6tables
(In reply to comment #38) > bent: > I'm grasping at straws here now, but let's try a few bits. > You have chosen a good straw - it works now :-) Please see the uploaded rc.log.
Okay, so we're stuck with a tough decision here. There is a circular dependency between: net.eth0, logger, iptables/ip6tables. iptables has 'use logger' AND 'before net'. => This implies an order of: logger, iptables, net syslog-ng with network logging has 'need net' or 'need net.eth0' => This implies an order of: net, logger So we're faced with a dilemma in solving the ordering: - If we start iptables after net, there might be a moment with unfirewalled services - If we start logger after net or after iptables, some log messages might be lost (syslog-ng does not buffer very long without the non-open-source plugin). Complete list of possible orders: logger, iptables, net - outgoing messages lost until net comes up iptables, logger, net - outgoing messages lost until net comes up logger, net, iptables - if logger opens a listening port, unfirewalled net, logger, iptables - if logger opens a listening port, unfirewalled net, iptables, logger - messages from iptables lost AND unfirewalled iptables, net, logger - messages from iptables lost Which is the lesser evil? My vote is for an order of 'iptables, net, logger' until such time as syslog-ng can buffer, and then 'logger, iptables, net'.
I concur with you. The best order in my opinion is also: 'iptables, net, logger' iptables must in any case come before net.
I tend to agree, and the way to fix this would be for the iptables and ip6tables init scripts to drop the "use logger" line from their depend function. The depend function for these scripts would then just have the line "before net".
Since we agree that this is an issue with the iptables/ip6tables init scripts and not with OpenRC itself, I am marking this bug to be a duplicate of the original bug for this same issue. *** This bug has been marked as a duplicate of bug 387751 ***
