After updating to systemd-249.11 systemd-resolve breaks using the default configuration with "DNSSEC validation failed: missing-key". Reproducible: Always Steps to Reproduce: 1. merge sys-apps/systemd-249.11 2. run `resolvectl query gentoo.org` 3. dns fails Actual Results: After emerging systemd to the latest stable version dns slowly starts breaking down. ``` $ resolvectl query gentoo.org gentoo.org: resolve call failed: DNSSEC validation failed: missing-key ``` ``` $ journalctl --unit systemd-resolved Mar 29 09:59:14 hostname systemd-resolved[1386]: [🡕] DNSSEC validation failed for question . IN DNSKEY: missing-key Mar 29 09:59:14 hostname systemd-resolved[1386]: [🡕] DNSSEC validation failed for question com IN DS: missing-key Mar 29 09:59:14 hostname systemd-resolved[1386]: [🡕] DNSSEC validation failed for question com IN DNSKEY: missing-key Mar 29 09:59:14 hostname systemd-resolved[1386]: [🡕] DNSSEC validation failed for question mozilla.com IN DS: missing-key .... Logs continue in this fashion ``` ``` $ grep -vE '^#' /etc/systemd/resolved.conf [Resolve] ``` Expected Results: DNS resolving continues to work after updating systemd ## Workaround: Adding: "DNSSEC=false" to "/etc/systemd/resolved.conf" resulted in a working system. ## Info sys-apps/systemd use flags: +abi_x86_32 +acl +cgroup-hybrid +gcrypt +kmod +lz4 +pam +pcre +policykit +resolvconf +seccomp +sysv-utils +zstd
emerge.log shows that it merged glibc right before systemd, which I now think is the issue... ``` emerge.log: Tue Mar 29 09:59:17 2022 >>> sys-libs/glibc-2.34-r10 Tue Mar 29 09:59:23 2022 >>> sys-devel/binutils-config-5.4.1 Tue Mar 29 10:00:44 2022 >>> sys-apps/systemd-249.11 Tue Mar 29 10:00:51 2022 >>> dev-python/flit_core-3.7.1 systemd-resolve log: Mar 29 09:59:14 hostname systemd-resolved[1386]: [🡕] DNSSEC validation failed for question ``` After trying 250.4, and 9999, both had the same problems, so i reverted back to 249.9 and rebooted. Same issue. DNSSEC failing notifications. The timings nearly match up exactly for glibc issues rather than the systemd install... (see attached emerge.log)
Created attachment 768103 [details] emerge log with exact timings
Hi I have the very same behaviour of OP. It seems related to upstream systemd issue 10579 (can't post links) which is present since 2018. I suspected that glibc switched from its own dns resolver to systemd in 2.34 but I found nothig in the release notes.
Based on comment 1, this is probably not a regression in systemd, but rather some interaction with a library upgrade.
Is this still a problem with recent systemd releases?
Hi Mike, Thanks for reminding me. (In reply to Mike Gilbert from comment #5) > Is this still a problem with recent systemd releases? I masked all systemd versions after 249.9 so i could have a working system. I have since updated to systemd-253.3-r1 and it successfully works without the workaround listed above.
I spoke too soon, but I have successfully replicated the issue on my system with the latest systemd version. (systemd-253.6) The problem seems to relate to using a DNS resolver that is "outside" my network, that does not support DNSSEC. resolvectl doesn't seem to figure out that it isn't supported and as such continues to try? (I'm guessing here) Replication: 1. Pick a random DNS server from OpenNIC: servers.opennic.org 2. Set it as your DNS resolver: sudo resolvectl dns enp0 168.138.8.38 3. Show the status: resolvectl status enp0 ``` Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6 Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=allow-downgrade/supported DNS Servers: 168.138.8.38 ``` 3.1 See the DNSSEC=allow-downgrade/supported 4. Query Something: resolvectl query bugs.gentoo.org 4.1 `bugs.gentoo.org resolve call failed: DNSSEC validation failed: missing-key` 5. Check the status again: resolvectl status enp0 5.1 Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=allow-downgrade/supported 5.2 Still shows as supported, even though the server actually doesn't support it 6. Change back to your local router dns: sudo resolvectl dns enp0 192.168.0.1 && resolvectl status enp0 6.1 it will show DNSSEC=allow-downgrade/supported until a DNS request is done 7. resolvectl query bugs.gentoo.org && resolvectl status enp0 7.1 correctly resolves and shows that DNSSEC=allow-downgrade/unsupported Proof: Use OpenDNS' 208.67.222.222 and you can see that dnssec passes and no issues arise. ---------- Work around: don't use a Non-DNSSEC enabled external resolver.
I have a similar issue on the local network of the science faculty of my university. This network has some DNS server to resolve local intranet addresses , it does not support DNSSEC yet systemd-resolved insists on trying and failing to use DNSSEC anyway. I have worked around the problem by changing the default DNSSEC=allow-downgrade setting to DNSSEC=no. But this is not really a good solution.
There was a merge in the area of systemd-resolved DNSSEC handling which claims to fix some issues like this: https://github.com/systemd/systemd/issues/24171 https://github.com/systemd/systemd/pull/31827
commit 1b646e8e63408abcdbf131ace4af9bb80ed5e29a Author: Sam James <sam@gentoo.org> Date: Mon Apr 29 18:26:28 2024 +0100 sys-apps/systemd: backport dnssec patch to 255 Bug: https://github.com/systemd/systemd/issues/32531 Signed-off-by: Sam James <sam@gentoo.org>
(In reply to Sam James from comment #10) > commit 1b646e8e63408abcdbf131ace4af9bb80ed5e29a > Author: Sam James <sam@gentoo.org> > Date: Mon Apr 29 18:26:28 2024 +0100 > > sys-apps/systemd: backport dnssec patch to 255 > > Bug: https://github.com/systemd/systemd/issues/32531 > Signed-off-by: Sam James <sam@gentoo.org> On my end systemd-resolved in 255.5-r1 still fails to resolve some pages if the dnssec setting is set to "allow-downgrade". I think we might also need to backport this patch: https://github.com/systemd/systemd/pull/32598
(In reply to Andrew Ammerlaan from comment #11) > (In reply to Sam James from comment #10) > > commit 1b646e8e63408abcdbf131ace4af9bb80ed5e29a > > Author: Sam James <sam@gentoo.org> > > Date: Mon Apr 29 18:26:28 2024 +0100 > > > > sys-apps/systemd: backport dnssec patch to 255 > > > > Bug: https://github.com/systemd/systemd/issues/32531 > > Signed-off-by: Sam James <sam@gentoo.org> > > On my end systemd-resolved in 255.5-r1 still fails to resolve some pages if > the dnssec setting is set to "allow-downgrade". > > I think we might also need to backport this patch: > https://github.com/systemd/systemd/pull/32598 Yeah, got it queued already - thanks though!
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=a2f26b71bd5e7b811ad0a085459c32ac149239b8 commit a2f26b71bd5e7b811ad0a085459c32ac149239b8 Author: Sam James <sam@gentoo.org> AuthorDate: 2024-05-05 15:41:52 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2024-05-05 15:43:30 +0000 sys-apps/systemd: backport more dnssec fixes to 255.5 Followup to 1b646e8e63408abcdbf131ace4af9bb80ed5e29a. Bug: https://bugs.gentoo.org/836341 Signed-off-by: Sam James <sam@gentoo.org> sys-apps/systemd/files/255-dnssec-2.patch | 48 +++ sys-apps/systemd/files/255-dnssec-3.patch | 32 ++ sys-apps/systemd/files/255-dnssec.patch | 8 +- sys-apps/systemd/systemd-255.5-r2.ebuild | 532 ++++++++++++++++++++++++++++++ 4 files changed, 617 insertions(+), 3 deletions(-)
Awesome, -r2 resolves the DNSSEC problems on my end
yay! Tim, could you confirm if it's OK for you too?
Installed: ``` sys-apps/systemd-255.5-r2::gentoo was built with the following: USE="acl cgroup-hybrid dns-over-tls gcrypt kernel-install kmod lz4 openssl pam pcre policykit resolvconf seccomp sysv-utils zstd -apparmor -audit -boot -cryptsetup -curl -elfutils -fido2 -gnutls -homed -http -idn -importd -iptables -lzma -pkcs11 -pwquality -qrcode -secureboot (-selinux) (-split-usr) -test -tpm -ukify -vanilla -xkb" ABI_X86="32 (64) (-x32)" PYTHON_SINGLE_TARGET="python3_11 -python3_10 -python3_12" ``` It does not fix my problem when using a server outside my local network which doesn't support DNSSEC. resolvectl dns enp0 80.152.203.134 resolvectl query bugs.gentoo.org > bugs.gentoo.org resolve call failed: DNSSEC validation failed: missing-key resolvectl dns enp0 37.252.191.197 resolvectl query bugs.gentoo.org > bugs.gentoo.org resolve call failed: DNSSEC validation failed: missing-key resolvectl dns enp0 94.247.43.254 resolvectl query bugs.gentoo.org > bugs.gentoo.org resolve call failed: DNSSEC validation failed: missing-key resolvectl dns enp0 1.1.1.1 resolvectl query bugs.gentoo.org > bugs.gentoo.org: 140.211.166.174 -- link: enp0
