dhcp-3.0_p2 works fine as root:root, but dhcp-3.0_p2-r2 is stable now and runs as dhcp:dhcp. Even if i manualy set the permissions on /var/lib/dhcp, they change back to root:root and the hdcpd process dies after the first new lease that needs to be written to the disk. Oct 23 10:12:12 [dhcpd] Can't create new lease file: Permission denied I changed bak to dhcp-3.0_p2 and everything works again.
I had the same problem with dhcp-3.0_p2-r2 and I had to run the command ebuild /var/db/pkg/net-misc/dhcp-3.0_p2-r2/dhcp-3.0_p2-r2.ebuild config to build the chroot jail and set the appropriate permissions. Then you can restart the dhcp service and everything will run as normal.
A fix is in the works. Hang on.
Fixed in cvs.
Whatever is causing the bug it's still there: granted, with the new ebuild, the initial permissions are set correctly to dhcp:dhcp, but as soon as you do /etc/init.d/dhcp start as root, they get reset to root:root Jeroen
The only thing in the startup script that changes permissions is a chown to dhcp:dhcp on the leases file. Can you verify this?
Ok. Here's what I've done. Fresh install of dhcp-3.0_p2-r2 (manually removed /var/lib/dhcp before emergeing) and checked the permissions: drwxr-xr-x 2 dhcp dhcp 1.0K Oct 31 08:40 dhcp drwxr-xr-x 2 dhcp dhcp 1024 Oct 31 08:40 . drwxr-xr-x 16 root root 1024 Oct 31 08:40 .. -rw-r--r-- 1 root root 0 Oct 31 08:40 .keep Next, I edited /etc/init.d/dhcp to stop executing just after checkconfig() is run by commenting out these lines: # start-stop-daemon --start --quiet --exec /usr/sbin/dhcpd \ # -- -user dhcp -group dhcp ${DHCPD_OPTS} \ # ${CHROOT:+-chroot ${CHROOT}} ${IFACE} Permissions on /var/lib/dhcp/dhcp.leases: drwxr-xr-x 2 dhcp dhcp 1024 Oct 31 08:52 . drwxr-xr-x 16 root root 1024 Oct 31 08:40 .. -rw-r--r-- 1 root root 0 Oct 31 08:40 .keep -rw-r--r-- 1 dhcp dhcp 0 Oct 31 08:52 dhcpd.leases Then I uncommented start-stop-daemon ... and restart: drwxr-xr-x 2 dhcp dhcp 1024 Oct 31 08:54 . drwxr-xr-x 16 root root 1024 Oct 31 08:40 .. -rw-r--r-- 1 root root 0 Oct 31 08:40 .keep -rw-r--r-- 1 root root 467 Oct 31 08:54 dhcpd.leases -rw-r--r-- 1 dhcp dhcp 0 Oct 31 08:52 dhcpd.leases~ As you can see the leases file gets reset. Since I'm also using dnsmasq, which uses the leases file as well, I thought maybe that was causing the problem so I set dnsmasq to run as dhcp:dhcp but no luck. The leases file still gets set to root:root and I'm out of ideas to try. As far as I can tell the script correctly sets the permissions but then either start-stop-daemon or dhcpd itself changes them back to the uid/gid of the user executing the script although I wouldn't know why, neither are suid root. Jeroen
Created attachment 20205 [details] Mail to dhcp-hackers@isc.org
Created attachment 20206 [details] Temporary (headache-) workaround
I have the same problem. I did "strace -f /usr/sbin/dhcpd -user dhcp -group dhcp eth0,br0" and looked carefully. I think the problem is: .... fsync(5) = 0 unlink("/var/lib/dhcp/dhcpd.leases~") = 0 link("/var/lib/dhcp/dhcpd.leases", "/var/lib/dhcp/dhcpd.leases~") = 0 rename("/var/lib/dhcp/dhcpd.leases.1067901142", "/var/lib/dhcp/dhcpd.leases") = 0 socket(PF_INET, SOCK_DGRAM, IPPROTO_UDP) = 6 ioctl(6, 0x8912, 0xbfffe998) = 0 ioctl(6, 0x8912, 0xbfffe998) = 0 ... Error report sent to dhcp-hackers@isc.org, see attached file. My actual (headache-) workaround is attached, I do not start dhcpd as dhcp:dhcp.
Couple of things: First, the paranoia patch used in the dhcp ebuild is a 3rd party project and not maintained by isc folks AFAIK. So I'm not sure they'll be able to help. Second, can you see at what point dhcpd drops its privs (look for setuid or setuid32 or somesuch). If it drops privs before it creates the temporary lease file then it doesn't make sense for the filed to be owned by root -- it should be owned by dhcp. And as I've said before, this is working flawlessly for me: jimbo local # ls -l /data/dhcp/var/lib/dhcp/ total 32 -rw-r--r-- 1 dhcp dhcp 13477 Nov 3 16:18 dhcpd.leases -rw-r--r-- 1 dhcp dhcp 15142 Nov 3 16:18 dhcpd.leases~ jimbo local # emerge -s 'dhcp$' Searching... [ Results for search key : dhcp$ ] [ Applications found : 2 ] * net-misc/dhcp Latest version available: 3.0_p2-r2 Latest version installed: 3.0_p2-r2 Size of downloaded files: 850 kB Homepage: http://www.isc.org/products/DHCP Description: ISC Dynamic Host Configuration Protocol.
Created attachment 20276 [details] Output generated by strace
I did the same strace as Thomas, and attached the output file, but for brevity I grepped on uid|dhcpd.leases with this result: 20122 geteuid32() = 0 20122 geteuid32() = 0 20122 open("/var/lib/dhcp/dhcpd.leases", O_RDONLY) = 5 20122 open("/var/lib/dhcp/dhcpd.leases", O_WRONLY|O_APPEND|O_CREAT, 0666) = 5 20122 open("/var/lib/dhcp/dhcpd.leases.1068002955", O_WRONLY|O_CREAT|O_TRUNC, 0664) = 5 20122 unlink("/var/lib/dhcp/dhcpd.leases~") = 0 20122 link("/var/lib/dhcp/dhcpd.leases", "/var/lib/dhcp/dhcpd.leases~") = 0 20122 rename("/var/lib/dhcp/dhcpd.leases.1068002955", "/var/lib/dhcp/dhcpd.leases") = 0 20123 setuid32(0x3f3 <unfinished ...> 20123 <... setuid32 resumed> ) = 0 As you can see, dhcpd first goes through the process of creating the backup leases file before switching to uid 0x3f3 (dhcp's uid). Jeroen
Ok, I confirm. It does do the setuid after initializing the leases file. The code also shows that db_start() is done before PARANOIA does it's voodoo. My question now is: Does having the file descriptor open before the privileges are dropped allow the file to be written. My guess is yes, so we're back to square-one. I don't know why people are seeing crashes. Unless dhcpd closes and reopens the file at some point. Still researching this.
Ok, found it. In db.c the function new_lease_file() will close the old FD and open a new one. This function is called when any write operation detects that the lease file is corrupt for some reason. It will also open a new lease file when nothing has been written for an hour.
Would you be able to try the experimental 3.0_p2-r3 version I just committed? It contains a patch to set proper ownership on the leases file after it has been opened (and before the privileges are dropped). Thanks! In the mean time, I'm going to send this patch to the Paranoia-patch developer to get his opinion on the matter.
It works for me now. Even manually chowning the leases file to root:root doesn't bother dhcp any more. Thanks a lot. Jeroen
Bah! What works? -r3 or -r2? Bart or Tom can you try and report back? Thanks.
Huh, right - now dhcp-3.0_p2-r3 works fine ;-)) Thank you very much... keep up the good work. Tom
Not so fast. I just got this reply from the paranoia patch creater (to whom I'm eternaly grateful -- thanks!) I also checked the source, by the way, and there is only one place the error message "Can't create new lease file:" can be generated. This place is the standard function for rewriting the lease file, and it only occurs if the daemon can't open a new lease file for writing. I can only conclude, then, that the user's lease directory was not writable to his dhcp user. I also checked to see if there are any other open(2)-related calls that could possibly reopen the lease file for writing. There are none, so changing the file ownership is unnecessary, unless a user sets the sticky bit on his lease directory for some reason. If he does, he should know what he's breaking. To ensure that another patch hadn't introduced this functionality, i checked over gentoo's patch set for dhcp-3.0_p2-r2.ebuild. They don't relate to leases or the lease file, so they wouldn't cause this problem. Let me know if you want me to update the bug log with a statement. ari ----------------------------------------------------------------------- So it seems that there's no reason to have my patch that chowns. Would you mind terribly trying the -r2 version again and making sure that the /var/lib/dhcp directory is owned by dhcp:dhcp _and_ that it doesn't crash when the lease file needs to be recreated? Thanks! Much obliged.
Hm, that's strange. After "downemergeing" from dhcp-3.0_p2-r3 to dhcp-3.0_p2-r2 it works. /var/lib/dhcp/ has dhcp:dhcp. First start: /var/lib/dhcp/dhcpd.leases has dhcp:dhcp, starting dhcp changes to root:root. Second start: /var/lib/dhcp/dhcpd.leases has then root:root, and the start writes the lease-file again. I waited long enough, that /var/log/everything/current says me, that there were DHCPREQUEST's and DHCPACK's. So, "Permission dienied" was not written. It works, (but what was the error?) I deleted /var/lib/dhcp, re-emerged again, but /var/lib/dhcp has dhcp:dhcp correctly. Hmm.
Thank you for your deep analysis. Looks like I will leave the -r2 version and remove the -r3 version based on your discoveries. My guess is that you might have emerged an "old" -r2 version that didn't do the chown as part of pkg_postinst() and your /var/lib/dhcp dir remained owned by root:root. Things should be better now. Thanks again! I'll close the bug as fixed.
Uuups. Not so fast! I discovered exactly the same problem with -r2. I am currently testing -r3 also and it works until now perfectly. Please look at http://forums.gentoo.org/viewtopic.php?t=103690&highlight= there are others with this problem, too. Unfortunately after emerging -r3 I cannot say, what owner my /var/lib/dhcp directory had before, but if there where more than oner -r2 releases it would be good anyway to bump the version! I will have a look on monday at a "untainted" Gentoo system with a -r2 release!
please, put back -r3. -r2 stops working after a while, -r3 fixes the issue.
According to the author of the paranoia patch, what -r3 was doing is unnecessary. Can you post the results of ls -la /var/lib/dhcp? Thanks.
Also, do you folks have any other security systems in effect? Grsec or some such?
-r3 output # ll /chroot/dhcp/var/lib/dhcp/ total 8 -rw-r--r-- 1 dhcp dhcp 2407 Nov 9 14:57 dhcpd.leases -rw-r--r-- 1 dhcp dhcp 3177 Nov 9 14:24 dhcpd.leases~ -r2 output # ll /chroot/dhcp/var/lib/dhcp/ total 8 -rw-r--r-- 1 root root 2407 Nov 9 14:57 dhcpd.leases -rw-r--r-- 1 dhcp dhcp 3177 Nov 9 14:24 dhcpd.leases~ No grsec here, only chroot.
ls -la please.
-r3 (what I have installed now) zeddmore /home/nbensa # ls -la /chroot/dhcp/var/lib/dhcp/ total 9 drwxr-xr-x 2 dhcp dhcp 136 Nov 9 15:27 . drwxr-xr-x 3 dhcp dhcp 72 Oct 30 03:28 .. -rw-r--r-- 1 dhcp dhcp 0 Oct 30 03:28 .keep -rw-r--r-- 1 dhcp dhcp 1242 Nov 9 15:27 dhcpd.leases -rw-r--r-- 1 dhcp dhcp 3182 Nov 9 15:27 dhcpd.leases~ same for -r2 except dhcpd.leases is root:root
Reopening yet again. Many appologies for the hassles this has caused. I really want to get to the bottom of this. Here is the explanation from the author: The new_lease_file() function creates a new lease file, and then replaces the original lease file with it (see the newfname argument to the open(2) call in that function). The rename(2) function handles the unlinking of the original lease database, and the renaming of its replacement, atomically, from within the kernel. All normal unix permissioning applies, so the unlink won't fail. The fdopen(3) function isn't relevant, since it only assoicates a FILE stream with the existing descriptor. At the request of the author, we'll need the following: # emerge -C dhcp # unmerge dhcp # mv /var/lib/dhcp /tmp/dhcp.old # backup your current dhcp leases directory ^---- if you run dhcp under a chroot, adjust this directory accordingly # emerge --sync && emerge dhcp # re-emerge the latest dhcp server # script /tmp/dhcp.trace # start tracing commands and output # portageq best_version / dhcp # make sure we have to correct version # emerge --info # record this stuff into the trace file # ls -la /var/lib/dhcp # checking initial permissions on this dir ^---- if you run dhcp under a chroot, adjust this directory accordingly # strace -f -s 1024 -e trace=open,close,rename,chroot \ > start-stop-daemon --start --quiet --exec /usr/sbin/dhcpd \ > -- -q -pf /var/run/dhcp/dhcpd.pid -user dhcp -group dhcp \ > eth0 ^ ^------ adjust to your configuration `------- if you run dhcp under a chroot, add -chroot <dir> here Run this until it crashes, then press ^C and ^D to end script. After that, post your resulting /tmp/dhcp.trace file. I realize this is quite a big request, and I really appreciate your time in investigating this. Hopefully this will help us get to the bottom of this nagging problem. Thanks again.
To me happend the sam as to Jeroen. After reemergeing -r2 I could not observe the problem anymore till now. Unfortunately I forgot, that my other "untainted" system did not have the dhcp so I have no comparison to the original -r2 state.
Max, I haven't looked into this permissions error, because I haven't experienced it, but I did find a number of issues with the ebuild. Firstly, you have a typo in pkg_config(): chown -R dhcp:dhcp "${CHROOT}/var/lib" "${CHROOT}/var/lib" I think for the second /var/lib you mean /var/run. Secondly, since you are using -DEARLY_CHROOT, unless you make a ${CHROOT}/dev/log and tell syslog to look at it, you disable logging. This should be done like with: pkg_config(): > mkdir "${CHROOT}/dev" /etc/conf.d/sysklogd: SYSLOGD="-a ${CHROOT}/dev/log" Obviously, you'll need some clever scripting to add that option to the sysklogd config file. If you would like me to submit an ebuild with these changes, please let me know.
I cleaned up the whole chroot setup process. It better ensures permissions and directories and creates a /dev/log in the chroot environment. I recommend that users with problems, rm -rf ${CHROOT} and try setting it up again. See attached.
Created attachment 20916 [details] dhcp ebuild with updated pkg_config()
You're assuming that the user is running sysklogd, which is not always true (I personally like syslog-ng). I think the best way to do this, is to just add a note at pkg_config() that informs the user to configure their logger accordingly. I did add the creation of /dev in the chroot and fixed the copy/paste error for /var/lib. Thanks!
I think the severity should be bumped to major or critical as this is a show stopper, a crashing dhcpd is quite a big issue (unless you are using daemontools). Also I think everyone here can agree that dhcpd does in fact crash, so why not bumb this back to unstable?
I re-added the ebuild with the chown patch. Please give this a try and report back.
Lets try closing again. If problems persist, please reopen.
I am still seeing this problem on my dhcp-3.0_p2-r2 on my stable server. I tried clearing the /var/lib/dhcp directory manually but dhcpd.leases always returns to root:root onership. I upgraded to dhcp-3.0_p2-r3 and the problem went away. I have had this problem for some time but I have dhcp under daemontools control so it was more of an inconvenience than a critical system problem. Perhaps dhcp-3.0_p2-r3 should be moved to stable.
