Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 836341 - sys-apps/systemd-249.11 update resulted in forced DNSSEC validation and everything failing
Summary: sys-apps/systemd-249.11 update resulted in forced DNSSEC validation and every...
Status: UNCONFIRMED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: AMD64 Linux
: Normal major with 1 vote (vote)
Assignee: Gentoo systemd Team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-03-29 05:04 UTC by Tim
Modified: 2024-01-09 00:05 UTC (History)
4 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
emerge log with exact timings (emerge.log,2.66 KB, text/x-log)
2022-03-29 06:26 UTC, Tim
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Tim 2022-03-29 05:04:55 UTC
After updating to systemd-249.11 systemd-resolve breaks using the default configuration with "DNSSEC validation failed: missing-key".

Reproducible: Always

Steps to Reproduce:
1. merge sys-apps/systemd-249.11
2. run `resolvectl query gentoo.org`
3. dns fails
Actual Results:  
After emerging systemd to the latest stable version dns slowly starts breaking down.

```
$ resolvectl query gentoo.org

gentoo.org: resolve call failed: DNSSEC validation failed: missing-key
```

```
$ journalctl --unit systemd-resolved

Mar 29 09:59:14 hostname systemd-resolved[1386]: [🡕] DNSSEC validation failed for question . IN DNSKEY: missing-key
Mar 29 09:59:14 hostname systemd-resolved[1386]: [🡕] DNSSEC validation failed for question com IN DS: missing-key
Mar 29 09:59:14 hostname systemd-resolved[1386]: [🡕] DNSSEC validation failed for question com IN DNSKEY: missing-key
Mar 29 09:59:14 hostname systemd-resolved[1386]: [🡕] DNSSEC validation failed for question mozilla.com IN DS: missing-key
....
Logs continue in this fashion
```

```
$ grep -vE '^#' /etc/systemd/resolved.conf
[Resolve]
```


Expected Results:  
DNS resolving continues to work after updating systemd

## Workaround:
Adding: "DNSSEC=false" to "/etc/systemd/resolved.conf" resulted in a working system.

## Info

sys-apps/systemd use flags:
+abi_x86_32
+acl
+cgroup-hybrid
+gcrypt
+kmod
+lz4
+pam
+pcre
+policykit
+resolvconf
+seccomp
+sysv-utils
+zstd
Comment 1 Tim 2022-03-29 06:25:34 UTC
emerge.log shows that it merged glibc right before systemd, which I now think is the issue...

```
emerge.log:

Tue Mar 29 09:59:17 2022 >>> sys-libs/glibc-2.34-r10
Tue Mar 29 09:59:23 2022 >>> sys-devel/binutils-config-5.4.1
Tue Mar 29 10:00:44 2022 >>> sys-apps/systemd-249.11
Tue Mar 29 10:00:51 2022 >>> dev-python/flit_core-3.7.1


systemd-resolve log: 

Mar 29 09:59:14 hostname systemd-resolved[1386]: [🡕] DNSSEC validation failed for question
```


After trying 250.4, and 9999, both had the same problems, so i reverted back to 249.9 and rebooted. Same issue. DNSSEC failing notifications.
The timings nearly match up exactly for glibc issues rather than the systemd install... (see attached emerge.log)
Comment 2 Tim 2022-03-29 06:26:18 UTC
Created attachment 768103 [details]
emerge log with exact timings
Comment 3 rodolfo 2022-04-07 05:30:11 UTC
Hi

I have the very same behaviour of OP.

It seems related to upstream systemd issue 10579 (can't post links) which is present since 2018.

I suspected that glibc switched from its own dns resolver to systemd in 2.34 but I found nothig in the release notes.
Comment 4 Mike Gilbert gentoo-dev 2022-04-08 23:09:43 UTC
Based on comment 1, this is probably not a regression in systemd, but rather some interaction with a library upgrade.
Comment 5 Mike Gilbert gentoo-dev 2023-07-22 20:37:15 UTC
Is this still a problem with recent systemd releases?
Comment 6 Tim 2023-07-23 05:27:26 UTC
Hi Mike,
Thanks for reminding me. 

(In reply to Mike Gilbert from comment #5)
> Is this still a problem with recent systemd releases?

I masked all systemd versions after 249.9 so i could have a working system. 

I have since updated to systemd-253.3-r1 and it successfully works without the workaround listed above.
Comment 7 Tim 2023-07-25 01:38:46 UTC
I spoke too soon, but I have successfully replicated the issue on my system with the latest systemd version. (systemd-253.6)

The problem seems to relate to using a DNS resolver that is "outside" my network, that does not support DNSSEC. 
resolvectl doesn't seem to figure out that it isn't supported and as such continues to try? (I'm guessing here)

Replication:

1. Pick a random DNS server from OpenNIC: servers.opennic.org
2. Set it as your DNS resolver: sudo resolvectl dns enp0 168.138.8.38
3. Show the status: resolvectl status enp0
```
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=allow-downgrade/supported
       DNS Servers: 168.138.8.38
```
3.1 See the DNSSEC=allow-downgrade/supported
4. Query Something: resolvectl query bugs.gentoo.org
4.1 `bugs.gentoo.org resolve call failed: DNSSEC validation failed: missing-key`
5. Check the status again: resolvectl status enp0
5.1 Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=allow-downgrade/supported
5.2 Still shows as supported, even though the server actually doesn't support it
6. Change back to your local router dns: sudo resolvectl dns enp0 192.168.0.1 && resolvectl status enp0
6.1 it will show DNSSEC=allow-downgrade/supported until a DNS request is done
7. resolvectl query bugs.gentoo.org && resolvectl status enp0
7.1 correctly resolves and shows that DNSSEC=allow-downgrade/unsupported


Proof: Use OpenDNS' 208.67.222.222 and you can see that dnssec passes and no issues arise.
----------

Work around: don't use a Non-DNSSEC enabled external resolver.