In the past couple weeks "tail -f /var/log/messages" on a couple machines on this LAN didn't output anything. In fact, it's been problematic on my file server for quite some time. This edit http://bpaste.net/show/64301/ was made to /etc/logrotate.d/syslog-ng and now /logrotate.debug reads: * You are attempting to run an openrc service on a * system which openrc did not boot. * You may be inside a chroot or you may have used * another initialization system to boot this system. * In this situation, you will get unpredictable results! * If you really want to do this, issue the following command: * touch /lib64/rc/init.d/softlevel syslog-ng reload failed! * You are attempting to run an openrc service on a * system which openrc did not boot. * You may be inside a chroot or you may have used * another initialization system to boot this system. * In this situation, you will get unpredictable results! * If you really want to do this, issue the following command: * touch /lib64/rc/init.d/softlevel syslog-ng reload failed! sys-apps/openrc-0.11.8 USE="ncurses unicode -debug -newnet -pam (-prefix) (-selinux) -static-libs" app-admin/logrotate-3.8.2 USE="acl (-selinux)" app-admin/syslog-ng-3.2.5 USE="pcre ssl tcpd -caps -hardened -ipv6 (-selinux) -spoof-source -sql" router ~ # uname -srm Linux 3.4.18 x86_64 This is my Linux router. Could be similar to bug 444652
Do you have a /run directory on the router? What message do you get if you run this command as root: /lib64/rc/sh/migrate-to-run.sh
(In reply to comment #1) > Do you have a /run directory on the router? > What message do you get if you run this command as root: > > /lib64/rc/sh/migrate-to-run.sh router ~ # ls -l /run/ total 0 drwxr-xr-x 2 root root 80 Nov 29 16:22 blkid drwxr-xr-x 2 root uucp 40 Dec 14 03:10 lock drwxr-xr-x 14 root root 360 Nov 7 09:51 openrc drwxr-xr-x 6 root root 140 Nov 7 09:51 udev router ~ # /lib64/rc/sh/migrate-to-run.sh * The OpenRC dependency data has already been migrated.
Do you have a directory, /lib64/rc/init.d? What is in that directory if you do?
Also, is /run a tmpfs?
Also, can you please attach the output from: /etc/init.d/syslog-ng -d reload Thanks, William
RL consumed my afternoon/evening, and there were run and pasted and literally not even read ... too exhausted atm. router ~ # ls -l /lib64/rc/init.d/ total 68 drwxr-xr-x 2 root root 4096 Jun 21 07:01 daemons -rw-r--r-- 1 root root 11 Jun 21 07:01 depconfig -rw-r--r-- 1 root root 14233 Jun 21 07:01 deptree drwxr-xr-x 2 root root 4096 Jun 21 07:01 exclusive drwxr-xr-x 2 root root 4096 Jun 21 07:01 failed drwxr-xr-x 2 root root 4096 Jun 21 07:01 hotplugged drwxr-xr-x 2 root root 4096 Jun 21 07:01 inactive drwxr-xr-x 2 root root 4096 Jun 21 07:01 options drwxr-xr-x 2 root root 4096 Jun 21 07:01 scheduled drwxr-xr-x 2 root root 4096 Jun 21 07:01 started drwxr-xr-x 2 root root 4096 Jun 21 07:01 starting drwxr-xr-x 2 root root 4096 Jun 21 07:01 stopping drwxr-xr-x 2 root root 4096 Jun 21 07:01 tmp drwxr-xr-x 2 root root 4096 Jun 21 07:01 wasinactive router ~ # df -hT Filesystem Type Size Used Avail Use% Mounted on rootfs rootfs 19G 3.0G 15G 17% / /dev/root ext4 19G 3.0G 15G 17% / tmpfs tmpfs 1.9G 156K 1.9G 1% /run cgroup_root tmpfs 10M 0 10M 0% /sys/fs/cgroup udev devtmpfs 10M 0 10M 0% /dev shm tmpfs 1.9G 0 1.9G 0% /dev/shm /dev/sda2 ext2 92M 20M 68M 23% /boot /dev/sda3 ext4 16G 983M 14G 7% /home router ~ # /etc/init.d/syslog-ng -d reload + _conf_d=/etc/init.d/../conf.d + _c=syslog-ng + '[' -n syslog-ng -a syslog-ng '!=' syslog-ng ']' + unset _c + sourcex -e /etc/init.d/../conf.d/syslog-ng.default + '[' -e = -e ']' + shift + '[' -e /etc/init.d/../conf.d/syslog-ng.default ']' + return 1 + sourcex -e /etc/init.d/../conf.d/syslog-ng + '[' -e = -e ']' + shift + '[' -e /etc/init.d/../conf.d/syslog-ng ']' + . /etc/init.d/../conf.d/syslog-ng ++ SYSLOG_NG_OPTS= + unset _conf_d + sourcex -e /etc/rc.conf + '[' -e = -e ']' + shift + '[' -e /etc/rc.conf ']' + . /etc/rc.conf ++ rc_shell=/sbin/sulogin ++ rc_logger=YES ++ unicode=YES ++ rc_sys= ++ rc_tty_number=12 + '[' Linux = Linux -a reload = start ']' + '[' -n '' ']' + sourcex /etc/init.d/syslog-ng + '[' /etc/init.d/syslog-ng = -e ']' + . /etc/init.d/syslog-ng ++ extra_commands=checkconfig ++ extra_started_commands=reload + unset _d + unset _f + '[' -n '' ']' + '[' -n reload ']' + '[' reload = depend ']' + for _cmd in describe start stop status '${extra_commands:-$opts}' '$extra_started_commands' '$extra_stopped_commands' + '[' describe = reload ']' + for _cmd in describe start stop status '${extra_commands:-$opts}' '$extra_started_commands' '$extra_stopped_commands' + '[' start = reload ']' + for _cmd in describe start stop status '${extra_commands:-$opts}' '$extra_started_commands' '$extra_stopped_commands' + '[' stop = reload ']' + for _cmd in describe start stop status '${extra_commands:-$opts}' '$extra_started_commands' '$extra_stopped_commands' + '[' status = reload ']' + for _cmd in describe start stop status '${extra_commands:-$opts}' '$extra_started_commands' '$extra_stopped_commands' + '[' checkconfig = reload ']' + for _cmd in describe start stop status '${extra_commands:-$opts}' '$extra_started_commands' '$extra_stopped_commands' + '[' reload = reload ']' ++ command -v reload + '[' reload = reload ']' + yesno + '[' -z '' ']' + return 1 + for _cmd in '$extra_started_commands' + '[' reload = reload ']' + verify_boot + '[' '!' -e /run/openrc/softlevel ']' + return 0 + service_started + unset _cmd + case $1 in ++ command -v reload_pre + '[' '' = reload_pre ']' + reload + '[' '!' -f /var/run/syslog-ng.pid ']' + checkconfig + '[' '!' -e /etc/syslog-ng/syslog-ng.conf ']' + syslog-ng -s -f /etc/syslog-ng/syslog-ng.conf + '[' 0 -eq 0 ']' + ebegin 'Reloading configuration and re-opening log files' * Reloading configuration and re-opening log files ... + start-stop-daemon --signal HUP --pidfile /var/run/syslog-ng.pid + eend 0 [ ok ] ++ command -v reload_post + '[' '' = reload_post ']' + shift + continue 2 + '[' -n '' ']' + exit 0
Your previous comment looks like a successful reload. So, is your cron script still giving you the message you put in the description?
(In reply to comment #7) > Your previous comment looks like a successful reload. > So, is your cron script still giving you the message you put in the > description? Whatever heuristic OpenRC employs to draw that conclusion appears to be very much broken. The purpose of Syslog-ng logrotate is so one doesn't have to manually reload it, eh?
(In reply to comment #8) > Whatever heuristic OpenRC employs to draw that conclusion appears to be very > much broken. It looks like it works if you run it manually, e.g. as root # /etc/init.d/syslog-ng reload. The next thing I need to see is a similar trace when /etc/init.d/syslog-ng is run from logrotate. If you first rm /logrotate.debug, then use something similar to this for /etc/logrotate.d/syslog-ng [1], you will get this information in /logrotate.debug. Once you do that, please attach /logrotate.debug once it has information in it. Thanks much for your I also have this set up now to see how it works here since I have been unable to reproduce your issue. Thanks for your patience and working with me to figure this out. [1] http://bpaste.net/show/64804
(In reply to comment #9) > (In reply to comment #8) > > Whatever heuristic OpenRC employs to draw that conclusion appears to be very > > much broken. > > It looks like it works if you run it manually, e.g. as root > > # /etc/init.d/syslog-ng reload. > > The next thing I need to see is a similar trace when /etc/init.d/syslog-ng > is run from logrotate. If you first rm /logrotate.debug, then use something > similar to this for /etc/logrotate.d/syslog-ng [1], you will get this > information in /logrotate.debug. Once you do that, please attach > /logrotate.debug once it has information in it. Thanks much for your > I also have this set up now to see how it works here since I have been > unable to reproduce your issue. Thanks for your patience and working with me > to figure this out. > > [1] http://bpaste.net/show/64804 I have done as you asked on router. However, /etc/logrotate.conf specifies weekly as the default. Today there was another /logrotate.debug entry identical to the previous ones, so unless I change weekly to daily we'll have to wait until Dec. 23. What do you recommend?
My previous post here was at 2012-12-16 14:58:46 CST -- that's Dec 16 02:58:46 This morning there is: router ~ # ls -l /var/log/messages* -rw------- 1 root root 0 Dec 16 03:10 /var/log/messages -rw------- 1 root root 1867601 Nov 25 03:10 /var/log/messages-20121125.gz -rw------- 1 root root 143098 Dec 2 03:10 /var/log/messages-20121202.gz -rw------- 1 root root 20 Dec 2 03:10 /var/log/messages-20121209.gz -rw------- 1 root root 166879 Dec 16 03:10 /var/log/messages-20121216.gz and the lines from /var/log/messages-20121216.gz since that epoch: Dec 16 02:58:04 router dhcpd: DHCPREQUEST for 192.168.100.5 from bc:ae:c5:6c:3c:97 via eth1 Dec 16 02:58:04 router dhcpd: DHCPACK on 192.168.100.5 to bc:ae:c5:6c:3c:97 via eth1 Dec 16 02:58:24 router dhcpd: DHCPREQUEST for 192.168.54.12 from 14:7d:c5:80:b4:86 via eth0 Dec 16 02:58:24 router dhcpd: DHCPACK on 192.168.54.12 to 14:7d:c5:80:b4:86 via eth0 Dec 16 03:02:23 router dhcpd: uid lease 192.168.100.13 for client bc:ae:c5:48:27:ef is duplicate on 192.168.100.0/24 Dec 16 03:02:23 router dhcpd: DHCPREQUEST for 192.168.100.6 from bc:ae:c5:48:27:ef via eth1 Dec 16 03:02:23 router dhcpd: DHCPACK on 192.168.100.6 to bc:ae:c5:48:27:ef via eth1 Dec 16 03:04:00 router dhcpd: DHCPREQUEST for 192.168.100.4 from 00:1b:21:b8:e2:f8 via eth1 Dec 16 03:04:00 router dhcpd: DHCPACK on 192.168.100.4 to 00:1b:21:b8:e2:f8 via eth1 Dec 16 03:05:57 router kernel: [3339530.694297] IN=eth1 OUT=eth2 MAC=68:05:ca:03:05:50:bc:ae:c5:6c:3c:97:08:00 SRC=192.168.100.5 DST=74.208.5.5 LEN=40 TOS=0x00 PREC=0x20 TTL=63 ID=0 DF PROTO=TCP SPT=57615 DPT=995 WINDOW=0 RES=0x00 RST URGP=0 Dec 16 03:06:03 router kernel: [3339537.169906] IN=eth2 OUT= MAC=f4:6d:04:e8:1d:d9:00:22:2d:5b:eb:eb:08:00 SRC=113.230.188.169 DST=66.208.231.133 LEN=56 TOS=0x00 PREC=0x20 TTL=244 ID=47229 PROTO=ICMP TYPE=11 CODE=0 [SRC=66.208.231.133 DST=112.175.124.5 LEN=40 TOS=0x00 PREC=0x00 TTL=1 ID=256 DF PROTO=TCP INCOMPLETE [8 bytes] ] Dec 16 03:09:26 router dhcpd: DHCPREQUEST for 192.168.100.20 from 6c:62:6d:f3:27:a8 (coolermaster) via eth1 Dec 16 03:09:26 router dhcpd: DHCPACK on 192.168.100.20 to 6c:62:6d:f3:27:a8 (coolermaster) via eth1 Dec 16 03:09:47 router dhcpd: Dynamic and static leases present for 192.168.54.16. Dec 16 03:09:47 router dhcpd: Remove host declaration baruch or remove 192.168.54.16 Dec 16 03:09:47 router dhcpd: from the dynamic address pool for 192.168.54.0/24 Dec 16 03:09:47 router dhcpd: DHCPREQUEST for 192.168.54.16 from a0:88:b4:54:33:04 via eth0 Dec 16 03:09:47 router dhcpd: DHCPACK on 192.168.54.16 to a0:88:b4:54:33:04 via eth0 Dec 16 03:10:01 router run-crons[25529]: (root) CMD (/etc/cron.daily/logrotate.cron) Dec 16 03:10:01 router /etc/init.d/syslog-ng[25537]: You are attempting to run an openrc service on a Dec 16 03:10:01 router /etc/init.d/syslog-ng[25538]: system which openrc did not boot. Dec 16 03:10:01 router /etc/init.d/syslog-ng[25539]: You may be inside a chroot or you may have used Dec 16 03:10:01 router /etc/init.d/syslog-ng[25540]: another initialization system to boot this system. Dec 16 03:10:01 router /etc/init.d/syslog-ng[25541]: In this situation, you will get unpredictable results! Dec 16 03:10:01 router /etc/init.d/syslog-ng[25543]: If you really want to do this, issue the following command: Dec 16 03:10:01 router /etc/init.d/syslog-ng[25544]: touch /lib64/rc/init.d/softlevel NB: There is no /logrotate.debug router ~ # ls -l /logrotate.debug ls: cannot access /logrotate.debug: No such file or directory
=app-admin/logrotate-3.8.3 was marked stable for amd64 yesterday. Can you reproduce this with last version?
(In reply to comment #12) > =app-admin/logrotate-3.8.3 was marked stable for amd64 yesterday. > > Can you reproduce this with last version? Just got to stop and check the LAN for other effected comps: server ~ # ls -l /var/log/messages* -rw------- 1 root root 0 Dec 16 03:10 /var/log/messages -rw------- 1 root root 3049 Nov 25 03:10 /var/log/messages-20121125.gz -rw------- 1 root root 2070 Dec 2 03:10 /var/log/messages-20121202.gz -rw------- 1 root root 20 Dec 2 03:10 /var/log/messages-20121209.gz -rw------- 1 root root 4390 Dec 16 03:10 /var/log/messages-20121216.gz I have not edited /etc/logrotate.d/syslog-ng on server. I did just update logrotate to app-admin/logrotate-3.8.3 My workstation also had the problem, but rotated properly this week. All 8 comps on this LAN have updated to logrotate-3.8.3 Now, on router do you want me to leave /etc/logrotate.d/syslog-ng with the edited file from williamh, or change it back to the original? And do you want me to change /etc/logrotate.conf to daily, or leave it at weekly and we see next weekend? So with router and server not presently logging, with what command do you want me to start logging again?
(In reply to comment #12) > =app-admin/logrotate-3.8.3 was marked stable for amd64 yesterday. > > Can you reproduce this with last version? The comp router was updated to logrotate-3.8.3 on 2012-12-17. In /etc/logrotate.conf weekly was changed to daily, and it has rotated every day since. (I just now changed it back to weekly.) router ~ # ls -l /var/log/messages* -rw------- 1 root root 90539 Dec 21 06:02 /var/log/messages -rw------- 1 root root 14598 Dec 18 06:10 /var/log/messages-20121218.gz -rw------- 1 root root 87692 Dec 19 03:10 /var/log/messages-20121219.gz -rw------- 1 root root 81947 Dec 20 03:10 /var/log/messages-20121220.gz -rw------- 1 root root 86460 Dec 21 03:10 /var/log/messages-20121221.gz The file /etc/logrotate.d/syslog-ng still had the version from William H in Comment 12 and /logrotate.debug now matches what was in Comment 6 AFAICT: http://bpaste.net/show/65756/ I have also changed that file to it's original: router ~ # cat /etc/logrotate.d/syslog-ng # $Header: /var/cvsroot/gentoo-x86/app-admin/syslog-ng/files/syslog-ng.logrotate,v 1.3 2008/10/15 20:46:12 mr_bones_ Exp $ # # Syslog-ng logrotate snippet for Gentoo Linux # contributed by Michael Sterrett # /var/log/messages { missingok # nocreate sharedscripts postrotate /etc/init.d/syslog-ng reload > /dev/null 2>&1 || true # /etc/init.d/syslog-ng -d reload >> /logrotate.debug 2>&1 || true endscript } Until next week's scheduled rotation...
(In reply to comment #14) > The file /etc/logrotate.d/syslog-ng still had the version from William H in > Comment 12 and /logrotate.debug now matches what was in Comment 6 AFAICT: > > http://bpaste.net/show/65756/ I was looking for the message in comment #0 to appear in the debug output. It doesn't, so it looks like you are successfully rotating your logs now.
I am closing this for now, but feel free to reopen if it comes back.
They are not rotating again... router ~ # ls -l /var/log/messages* -rw------- 1 root root 21984025 Jan 8 05:27 /var/log/messages -rw------- 1 root root 87692 Dec 19 03:10 /var/log/messages-20121219.gz -rw------- 1 root root 81947 Dec 20 03:10 /var/log/messages-20121220.gz -rw------- 1 root root 86460 Dec 21 03:10 /var/log/messages-20121221.gz -rw------- 1 root root 80402 Dec 23 03:10 /var/log/messages-20121223.gz router ~ # cat /etc/logrotate.d/syslog-ng # $Header: /var/cvsroot/gentoo-x86/app-admin/syslog-ng/files/syslog-ng.logrotate,v 1.3 2008/10/15 20:46:12 mr_bones_ Exp $ # # Syslog-ng logrotate snippet for Gentoo Linux # contributed by Michael Sterrett # /var/log/messages /var/log/dhcpd.log { missingok sharedscripts postrotate # /etc/init.d/syslog-ng reload > /dev/null 2>&1 || true stderr="$(/etc/init.d/syslog-ng -d reload 2>&1 >> /logrotate.stdout)" [ $? -eq 0 ] || echo "$stderr" >> /logrotate.stderr endscript } router ~ # ls -l /logrotate.stderr ls: cannot access /logrotate.stderr: No such file or directory
(In reply to comment #17) > router ~ # cat /etc/logrotate.d/syslog-ng > # $Header: > /var/cvsroot/gentoo-x86/app-admin/syslog-ng/files/syslog-ng.logrotate,v 1.3 > 2008/10/15 20:46:12 mr_bones_ Exp $ > /var/log/messages /var/log/dhcpd.log { > missingok > sharedscripts > postrotate > # /etc/init.d/syslog-ng reload > /dev/null 2>&1 || true > stderr="$(/etc/init.d/syslog-ng -d reload 2>&1 >> /logrotate.stdout)" > [ $? -eq 0 ] || echo "$stderr" >> /logrotate.stderr The stderr variable will always be empty because you are redirecting everything to /logrotate.stdout. I think what you want to do here is get rid of '>> /logrotate.stdout' in the stderr= line. > endscript > } > > router ~ # ls -l /logrotate.stderr > ls: cannot access /logrotate.stderr: No such file or directory See above for why this is empty. Once that is fixed, re-open and attach logrotate.stderr after the next run. Thanks, William
(In reply to comment #18) > (In reply to comment #17) > > router ~ # cat /etc/logrotate.d/syslog-ng > > # $Header: > > /var/cvsroot/gentoo-x86/app-admin/syslog-ng/files/syslog-ng.logrotate,v 1.3 > > 2008/10/15 20:46:12 mr_bones_ Exp $ > > /var/log/messages /var/log/dhcpd.log { > > missingok > > sharedscripts > > postrotate > > # /etc/init.d/syslog-ng reload > /dev/null 2>&1 || true > > stderr="$(/etc/init.d/syslog-ng -d reload 2>&1 >> /logrotate.stdout)" > > [ $? -eq 0 ] || echo "$stderr" >> /logrotate.stderr > > The stderr variable will always be empty because you are redirecting > everything to /logrotate.stdout. I think what you want to do here is get rid > of '>> /logrotate.stdout' in the stderr= line. > > > > endscript > > } > > > > router ~ # ls -l /logrotate.stderr > > ls: cannot access /logrotate.stderr: No such file or directory > > See above for why this is empty. Once that is fixed, re-open and attach > logrotate.stderr after the next run. > > Thanks, > > William I just can't agree with your statement concerning shell behavior, because workstation was setup just like router, as workstation also exhibited logs not rotating in the past. And on workstation I see: workstation ~ # ls -l /var/log/messages* -rw------- 1 root root 168674 Jan 9 08:24 /var/log/messages -rw------- 1 root root 30552 Dec 16 03:10 /var/log/messages-20121216.gz -rw------- 1 root root 65826 Dec 23 03:10 /var/log/messages-20121223.gz -rw------- 1 root root 37443 Dec 30 03:10 /var/log/messages-20121230.gz -rw------- 1 root root 34724 Jan 6 03:10 /var/log/messages-20130106.gz workstation ~ # ls -l /logrotate.stdout -rw-r--r-- 1 root root 126 Jan 6 03:10 /logrotate.stdout workstation ~ # cat /logrotate.stdout * Reloading configuration and re-opening log files ... [ ok ] * Reloading configuration and re-opening log files ... [ ok ] workstation ~ # cat /etc/logrotate.d/syslog-ng # $Header: /var/cvsroot/gentoo-x86/app-admin/syslog-ng/files/syslog-ng.logrotate,v 1.3 2008/10/15 20:46:12 mr_bones_ Exp $ # # Syslog-ng logrotate snippet for Gentoo Linux # contributed by Michael Sterrett # /var/log/messages { missingok sharedscripts postrotate # /etc/init.d/syslog-ng reload > /dev/null 2>&1 || true stderr="$(/etc/init.d/syslog-ng -d reload 2>&1 >> /logrotate.stdout)" [ $? -eq 0 ] || echo "$stderr" >> /logrotate.stderr endscript } workstation ~ # So logic says the syntax in /etc/logrotate.d/syslog-ng _does_ work properly. I believe what actually happens is that stderr is redirected to stdout and what was originally stdout is directed to /dev/null.
This bug isn't about logs not rotating; it is about OpenRC giving you the specific message in comment #0. I need to know if that message is still occurring. If it isn't, this issue is resolved. > missingok > sharedscripts > postrotate > # /etc/init.d/syslog-ng reload > /dev/null 2>&1 || true > stderr="$(/etc/init.d/syslog-ng -d reload 2>&1 >> /logrotate.stdout)" Redirection happens from left to right, so: 2>&1 redirect stderr to stdout. >> /logrotate.stdout redirects the combination of stderr and stdout (because of the 2>&1) to /logrotate.stdout That leaves for the assignment: stderr="", because the evaluation of the $(...) expression is empty. You need the 2>&1, but the >> /logrotate.stdout is not right if you want something to be assigned to the stderr variable at the beginning of the line: If you want to see this for yourself, try: x="$(date)" echo $x # This will print the date. then: x="$(date >> date.out)" echo $x # this prints nothing because x doesn't have a value. > [ $? -eq 0 ] || echo "$stderr" >> /logrotate.stderr This means that if the reload was successful, logrotate.stderr will not be created.
(In reply to comment #20) > This bug isn't about logs not rotating Close it then. That was my problem. It wasn't logging on three machines, actually. Then not rotating. Still not rotating on router. The fact that OpenRC gave such a brain dead message was peripheral.
