Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 836341 - sys-apps/systemd-249.11 update resulted in forced DNSSEC validation and everything failing
Summary: sys-apps/systemd-249.11 update resulted in forced DNSSEC validation and every...
Status: UNCONFIRMED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: AMD64 Linux
: Normal major with 1 vote (vote)
Assignee: Gentoo systemd Team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-03-29 05:04 UTC by Tim
Modified: 2024-05-06 00:29 UTC (History)
8 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
emerge log with exact timings (emerge.log,2.66 KB, text/x-log)
2022-03-29 06:26 UTC, Tim
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Tim 2022-03-29 05:04:55 UTC
After updating to systemd-249.11 systemd-resolve breaks using the default configuration with "DNSSEC validation failed: missing-key".

Reproducible: Always

Steps to Reproduce:
1. merge sys-apps/systemd-249.11
2. run `resolvectl query gentoo.org`
3. dns fails
Actual Results:  
After emerging systemd to the latest stable version dns slowly starts breaking down.

```
$ resolvectl query gentoo.org

gentoo.org: resolve call failed: DNSSEC validation failed: missing-key
```

```
$ journalctl --unit systemd-resolved

Mar 29 09:59:14 hostname systemd-resolved[1386]: [🡕] DNSSEC validation failed for question . IN DNSKEY: missing-key
Mar 29 09:59:14 hostname systemd-resolved[1386]: [🡕] DNSSEC validation failed for question com IN DS: missing-key
Mar 29 09:59:14 hostname systemd-resolved[1386]: [🡕] DNSSEC validation failed for question com IN DNSKEY: missing-key
Mar 29 09:59:14 hostname systemd-resolved[1386]: [🡕] DNSSEC validation failed for question mozilla.com IN DS: missing-key
....
Logs continue in this fashion
```

```
$ grep -vE '^#' /etc/systemd/resolved.conf
[Resolve]
```


Expected Results:  
DNS resolving continues to work after updating systemd

## Workaround:
Adding: "DNSSEC=false" to "/etc/systemd/resolved.conf" resulted in a working system.

## Info

sys-apps/systemd use flags:
+abi_x86_32
+acl
+cgroup-hybrid
+gcrypt
+kmod
+lz4
+pam
+pcre
+policykit
+resolvconf
+seccomp
+sysv-utils
+zstd
Comment 1 Tim 2022-03-29 06:25:34 UTC
emerge.log shows that it merged glibc right before systemd, which I now think is the issue...

```
emerge.log:

Tue Mar 29 09:59:17 2022 >>> sys-libs/glibc-2.34-r10
Tue Mar 29 09:59:23 2022 >>> sys-devel/binutils-config-5.4.1
Tue Mar 29 10:00:44 2022 >>> sys-apps/systemd-249.11
Tue Mar 29 10:00:51 2022 >>> dev-python/flit_core-3.7.1


systemd-resolve log: 

Mar 29 09:59:14 hostname systemd-resolved[1386]: [🡕] DNSSEC validation failed for question
```


After trying 250.4, and 9999, both had the same problems, so i reverted back to 249.9 and rebooted. Same issue. DNSSEC failing notifications.
The timings nearly match up exactly for glibc issues rather than the systemd install... (see attached emerge.log)
Comment 2 Tim 2022-03-29 06:26:18 UTC
Created attachment 768103 [details]
emerge log with exact timings
Comment 3 rodolfo 2022-04-07 05:30:11 UTC
Hi

I have the very same behaviour of OP.

It seems related to upstream systemd issue 10579 (can't post links) which is present since 2018.

I suspected that glibc switched from its own dns resolver to systemd in 2.34 but I found nothig in the release notes.
Comment 4 Mike Gilbert gentoo-dev 2022-04-08 23:09:43 UTC
Based on comment 1, this is probably not a regression in systemd, but rather some interaction with a library upgrade.
Comment 5 Mike Gilbert gentoo-dev 2023-07-22 20:37:15 UTC
Is this still a problem with recent systemd releases?
Comment 6 Tim 2023-07-23 05:27:26 UTC
Hi Mike,
Thanks for reminding me. 

(In reply to Mike Gilbert from comment #5)
> Is this still a problem with recent systemd releases?

I masked all systemd versions after 249.9 so i could have a working system. 

I have since updated to systemd-253.3-r1 and it successfully works without the workaround listed above.
Comment 7 Tim 2023-07-25 01:38:46 UTC
I spoke too soon, but I have successfully replicated the issue on my system with the latest systemd version. (systemd-253.6)

The problem seems to relate to using a DNS resolver that is "outside" my network, that does not support DNSSEC. 
resolvectl doesn't seem to figure out that it isn't supported and as such continues to try? (I'm guessing here)

Replication:

1. Pick a random DNS server from OpenNIC: servers.opennic.org
2. Set it as your DNS resolver: sudo resolvectl dns enp0 168.138.8.38
3. Show the status: resolvectl status enp0
```
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=allow-downgrade/supported
       DNS Servers: 168.138.8.38
```
3.1 See the DNSSEC=allow-downgrade/supported
4. Query Something: resolvectl query bugs.gentoo.org
4.1 `bugs.gentoo.org resolve call failed: DNSSEC validation failed: missing-key`
5. Check the status again: resolvectl status enp0
5.1 Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=allow-downgrade/supported
5.2 Still shows as supported, even though the server actually doesn't support it
6. Change back to your local router dns: sudo resolvectl dns enp0 192.168.0.1 && resolvectl status enp0
6.1 it will show DNSSEC=allow-downgrade/supported until a DNS request is done
7. resolvectl query bugs.gentoo.org && resolvectl status enp0
7.1 correctly resolves and shows that DNSSEC=allow-downgrade/unsupported


Proof: Use OpenDNS' 208.67.222.222 and you can see that dnssec passes and no issues arise.
----------

Work around: don't use a Non-DNSSEC enabled external resolver.
Comment 8 Andrew Ammerlaan gentoo-dev 2024-04-30 10:22:39 UTC
I have a similar issue on the local network of the science faculty of my university. This network has some DNS server to resolve local intranet addresses , it does not support DNSSEC yet systemd-resolved insists on trying and failing to use DNSSEC anyway.

I have worked around the problem by changing the default DNSSEC=allow-downgrade setting to DNSSEC=no. But this is not really a good solution.
Comment 9 John Helmert III archtester Gentoo Infrastructure gentoo-dev Security 2024-05-04 22:48:33 UTC
There was a merge in the area of systemd-resolved DNSSEC handling which claims to fix some issues like this:

https://github.com/systemd/systemd/issues/24171
https://github.com/systemd/systemd/pull/31827
Comment 10 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-05-05 15:38:21 UTC
commit 1b646e8e63408abcdbf131ace4af9bb80ed5e29a
Author: Sam James <sam@gentoo.org>
Date:   Mon Apr 29 18:26:28 2024 +0100

    sys-apps/systemd: backport dnssec patch to 255

    Bug: https://github.com/systemd/systemd/issues/32531
    Signed-off-by: Sam James <sam@gentoo.org>
Comment 11 Andrew Ammerlaan gentoo-dev 2024-05-05 15:43:24 UTC
(In reply to Sam James from comment #10)
> commit 1b646e8e63408abcdbf131ace4af9bb80ed5e29a
> Author: Sam James <sam@gentoo.org>
> Date:   Mon Apr 29 18:26:28 2024 +0100
> 
>     sys-apps/systemd: backport dnssec patch to 255
> 
>     Bug: https://github.com/systemd/systemd/issues/32531
>     Signed-off-by: Sam James <sam@gentoo.org>

On my end systemd-resolved in 255.5-r1 still fails to resolve some pages if the dnssec setting is set to "allow-downgrade".

I think we might also need to backport this patch: https://github.com/systemd/systemd/pull/32598
Comment 12 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-05-05 15:43:47 UTC
(In reply to Andrew Ammerlaan from comment #11)
> (In reply to Sam James from comment #10)
> > commit 1b646e8e63408abcdbf131ace4af9bb80ed5e29a
> > Author: Sam James <sam@gentoo.org>
> > Date:   Mon Apr 29 18:26:28 2024 +0100
> > 
> >     sys-apps/systemd: backport dnssec patch to 255
> > 
> >     Bug: https://github.com/systemd/systemd/issues/32531
> >     Signed-off-by: Sam James <sam@gentoo.org>
> 
> On my end systemd-resolved in 255.5-r1 still fails to resolve some pages if
> the dnssec setting is set to "allow-downgrade".
> 
> I think we might also need to backport this patch:
> https://github.com/systemd/systemd/pull/32598

Yeah, got it queued already - thanks though!
Comment 13 Larry the Git Cow gentoo-dev 2024-05-05 15:43:53 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=a2f26b71bd5e7b811ad0a085459c32ac149239b8

commit a2f26b71bd5e7b811ad0a085459c32ac149239b8
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2024-05-05 15:41:52 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2024-05-05 15:43:30 +0000

    sys-apps/systemd: backport more dnssec fixes to 255.5
    
    Followup to 1b646e8e63408abcdbf131ace4af9bb80ed5e29a.
    
    Bug: https://bugs.gentoo.org/836341
    Signed-off-by: Sam James <sam@gentoo.org>

 sys-apps/systemd/files/255-dnssec-2.patch |  48 +++
 sys-apps/systemd/files/255-dnssec-3.patch |  32 ++
 sys-apps/systemd/files/255-dnssec.patch   |   8 +-
 sys-apps/systemd/systemd-255.5-r2.ebuild  | 532 ++++++++++++++++++++++++++++++
 4 files changed, 617 insertions(+), 3 deletions(-)
Comment 14 Andrew Ammerlaan gentoo-dev 2024-05-05 19:07:47 UTC
Awesome, -r2 resolves the DNSSEC problems on my end
Comment 15 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-05-05 19:08:54 UTC
yay! Tim, could you confirm if it's OK for you too?
Comment 16 Tim 2024-05-06 00:29:54 UTC
Installed:

```
sys-apps/systemd-255.5-r2::gentoo was built with the following:
USE="acl cgroup-hybrid dns-over-tls gcrypt kernel-install kmod lz4 openssl pam pcre policykit resolvconf seccomp sysv-utils zstd -apparmor -audit -boot -cryptsetup -curl -elfutils -fido2 -gnutls -homed -http -idn -importd -iptables -lzma -pkcs11 -pwquality -qrcode -secureboot (-selinux) (-split-usr) -test -tpm -ukify -vanilla -xkb" ABI_X86="32 (64) (-x32)" PYTHON_SINGLE_TARGET="python3_11 -python3_10 -python3_12"
```

It does not fix my problem when using a server outside my local network which doesn't support DNSSEC. 

resolvectl dns enp0 80.152.203.134
resolvectl query bugs.gentoo.org
> bugs.gentoo.org resolve call failed: DNSSEC validation failed: missing-key
resolvectl dns enp0 37.252.191.197
resolvectl query bugs.gentoo.org
> bugs.gentoo.org resolve call failed: DNSSEC validation failed: missing-key
resolvectl dns enp0 94.247.43.254
resolvectl query bugs.gentoo.org
> bugs.gentoo.org resolve call failed: DNSSEC validation failed: missing-key
resolvectl dns enp0 1.1.1.1
resolvectl query bugs.gentoo.org
> bugs.gentoo.org: 140.211.166.174               -- link: enp0