Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 727262

Summary: app-misc/ca-certificates-20200601.3.53 breaks some Steam games
Product: Gentoo Linux Reporter: Tomasz Golinski <tomaszg>
Component: Current packagesAssignee: Gentoo's Team for Core System packages <base-system>
Status: RESOLVED INVALID    
Severity: normal CC: a.zuber, sam, techwolf.lupindo
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: Linux   
Whiteboard:
Package list:
Runtime testing required: ---
Attachments: output of ssldump (truncated as it was too large)

Description Tomasz Golinski 2020-06-05 21:55:27 UTC
After upgrading ca-certificates a launcher to The Elder Scrolls Online game fails to connect to its server and doesn't allow the game to start. Downgrading to ca-certificates-20190110.3.43 fixes the issue. The game runs with Proton, but that doesn't seem to be related.

I didn't notice any other game to be broken yet. 

Following instructions I got in bug 726412 I tried to see what was the reason but didn't succeed. The most likely culprit seems to be a connection to IP 198.20.198.110. I managed to get ssldump and tcpdump logs in case of good and bad ca-certificates but I have difficulty finding relevant information there. 

Here's a part of tcpdump:
22:36:33.025223 IP 192.168.2.1.36626 > 198.20.198.110.https: Flags [S], seq 4081917815, win 64240, options [mss 1460,sackOK,TS val 1240031938 ecr 0,nop,wscale 7], length 0
22:36:33.025541 IP 192.168.2.1.36628 > 198.20.198.110.https: Flags [S], seq 1484205525, win 64240, options [mss 1460,sackOK,TS val 1240031939 ecr 0,nop,wscale 7], length 0

I'll attach log from ssldump of a successful connection.

Reproducible: Always

Steps to Reproduce:
1. install app-misc/ca-certificates-20200601.3.53
2. install games-util/steam-launcher from steam-overlay
3. buy "The Elder Scrolls Online". Regretfully I don't know any free game which exhibits this problem yet.



/usr/lib/python3.7/site-packages/portage/package/ebuild/_config/KeywordsManager.py:70: UserWarning: /etc/portage/package.keywords is deprecated, use /etc/portage/package.accept_keywords instead
  UserWarning)
Portage 2.3.99 (python 3.7.7-final-0, default/linux/amd64/17.1/desktop, gcc-9.3.0, glibc-2.30-r8, 5.7.0-gentoo x86_64)
=================================================================
System uname: Linux-5.7.0-gentoo-x86_64-AMD_Ryzen_5_3600_6-Core_Processor-with-gentoo-2.6
KiB Mem:    32913008 total,   7100100 free
KiB Swap:          0 total,         0 free
Timestamp of repository gentoo: Thu, 04 Jun 2020 18:36:11 +0000
Head commit of repository gentoo: f39786e08ad68bec77964e418b1f3c4f106889f2

Timestamp of repository kde: Thu, 04 Jun 2020 14:35:12 +0000
Head commit of repository kde: 240d68494a98f47f4fc02da4d73f0984aefb33d9

Timestamp of repository steam-overlay: Mon, 01 Jun 2020 06:07:52 +0000
Head commit of repository steam-overlay: 45bd91ee3533a6273e54a353cee59cc8c2931d44

sh bash 5.0_p17
ld GNU ld (Gentoo 2.33.1 p2) 2.33.1
app-shells/bash:          5.0_p17::gentoo
dev-lang/perl:            5.30.1::gentoo
dev-lang/python:          2.7.18::gentoo, 3.7.7-r2::gentoo, 3.8.2-r2::gentoo
dev-util/cmake:           3.16.5::gentoo
sys-apps/baselayout:      2.6-r1::gentoo
sys-apps/openrc:          0.42.1::gentoo
sys-apps/sandbox:         2.18::gentoo
sys-devel/autoconf:       2.13-r1::gentoo, 2.69-r4::gentoo
sys-devel/automake:       1.13.4-r2::gentoo, 1.16.1-r1::gentoo
sys-devel/binutils:       2.33.1-r1::gentoo
sys-devel/gcc:            9.3.0-r1::gentoo, 10.1.0::gentoo
sys-devel/gcc-config:     2.2.1::gentoo
sys-devel/libtool:        2.4.6-r6::gentoo
sys-devel/make:           4.2.1-r4::gentoo
sys-kernel/linux-headers: 5.4-r1::gentoo (virtual/os-headers)
sys-libs/glibc:           2.30-r8::gentoo
Repositories:

gentoo
    location: /var/db/repos/gentoo
    sync-type: git
    sync-uri: https://anongit.gentoo.org/git/repo/sync/gentoo.git
    priority: -1000
    sync-git-verify-commit-signature: yes

kde
    location: /var/db/repos/kde
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/kde.git
    masters: gentoo

local
    location: /var/db/repos/local
    masters: gentoo

steam-overlay
    location: /var/db/repos/steam-overlay
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/steam-overlay.git
    masters: gentoo

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="@FREE"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -march=native -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-O2 -march=native -pipe"
DISTDIR="/var/cache/distfiles"
EMERGE_DEFAULT_OPTS="--quiet-build=n --jobs=20 --load-average=14 --keep-going"
ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -march=native -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs cgroup distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync"
FFLAGS="-O2 -march=native -pipe"
GENTOO_MIRRORS="http://ftp-stud.hs-esslingen.de/pub/Mirrors/gentoo/ http://ftp.snt.utwente.nl/pub/os/linux/gentoo http://mirror.leaseweb.com/gentoo/"
INSTALL_MASK="/usr/share/locale -/usr/share/locale/pl* -/usr/share/locale/en /usr/share/binutils-data/*/*/locale -/usr/share/binutils-data/*/*/locale/pl* -/usr/share/binutils-data/*/*/locale/en /usr/share/gcc-data/*/*/locale -/usr/share/gcc-data/*/*/locale/pl* -/usr/share/gcc-data/*/*/locale/en /usr/share/man/ -/usr/share/man/man* -/usr/share/man/pl* -/usr/share/man/whatis"
LANG="en_US.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j24 -l12"
PKGDIR="/var/cache/binpkgs"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="X a52 aac acoustid acpi alsa amd64 bash-completion berkdb branding bzip2 cairo cdda cdr cli consolekit cups dbus dri dts dvd dvdr emboss encode exif flac fortran gdbm gif gimp gpm gpu gtk gtk3 iconv icu jpeg lcms libnotify libtirpc lto mad mng mobi mp3 mp4 mpeg mtp multilib musepack musicbrainz ncurses nls nptl offensive ogg opencl opengl openmp opus pam pango pcre pdf pgo png policykit postscript ppds qt5 readline sane scanner sdl seccomp spell split-usr ssl startup-notification svg system-cairo system-harfbuzz system-icu system-jpeg system-jsoncpp system-libevent system-libvpx system-llvm system-sqlite tcpd tiff truetype udev udisks unicode upower usb userlocales vdpau vorbis vulkan webp wxwidgets x264 x265 xcb xml xmp xv xvid zlib" ABI_X86="64" ADA_TARGET="gnat_2018" ALSA_CARDS="intel8x0 emu10k1 virtuoso" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" CAMERAS="canon" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt sha sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput" KERNEL="linux" LCD_DEVICES="hd44780 curses text rawserial serialpos serialvfd" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LIRC_DEVICES="serial" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-2" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_7" PYTHON_TARGETS="python2_7 python3_7" RUBY_TARGETS="ruby25" SANE_BACKENDS="pixma" USERLAND="GNU" VIDEO_CARDS="amdgpu radeonsi radeon" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, LC_ALL, LINGUAS, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Comment 1 Tomasz Golinski 2020-06-05 21:58:16 UTC
Created attachment 643572 [details]
output of ssldump (truncated as it was too large)
Comment 2 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2020-06-06 06:47:00 UTC
There isn't any actual failure in this ssldump log, but ssldump is also really outdated.

198.20.198.110 serves up LIVE-SERVICES.ELDERSCROLLSONLINE.COM as the default certificate. It might have other responses but your logs don't contain enough detail to show SNI. That cert was issued by DigiCert, and is fine.

curl connects fine to that IP/hostname, with both versions of ca-certificates.

If you captured raw tcpdump packets, try play them through wireshark's text version "tshark -V", or attach here.

Can you also try ca-certificates-20190110.3.53? Just want to rule where the problem was introduced.
Comment 3 Tomasz Golinski 2020-06-06 18:02:17 UTC
The attached log was of successful connection. On "bad" version I don't see anything in ssldump.

ca-certificates-20190110.3.53 work fine for me.

I just tried experimenting with dumps again and I'm probably too stupid to understand what I'm doing. I tried again the bad certs and I don't see ANYTHING in tcpdump when the launcher starts other than my local network traffic, google dns queries and Steam talk on 155.133.230.50. However this Steam talk shows only when I start the game and stops long before launcher shows up. While it is "loading" I don't see any tcpdump activity at all.

In the case of good certs, after Steam chatter, I see a HTTP connection to 159.100.230.100 (which is Bethesda/Zenimax server, as expected) and it then moves to 198.20.198.110 via HTTPS (also Zenimax server). I don't see any packages sent to any of those IPs with bad cert package.

Here's an example of bad run:
tcpdump host not \( htpc or dns.google \)
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
19:45:23.520179 IP 192.168.2.1.46325 > 155.133.230.50.27025: Flags [P.], seq 3579209299:3579209404, ack 711374898, win 32406, options [nop,nop,TS val 3797168361 ecr 1181998594], length 105
19:45:23.520533 IP 192.168.2.1.46325 > 155.133.230.50.27025: Flags [P.], seq 105:393, ack 1, win 32406, options [nop,nop,TS val 3797168362 ecr 1181998594], length 288
19:45:23.524085 IP 155.133.230.50.27025 > 192.168.2.1.46325: Flags [.], ack 393, win 1027, options [nop,nop,TS val 1182007383 ecr 3797168361], length 0
19:45:23.772835 IP 155.133.230.50.27025 > 192.168.2.1.46325: Flags [P.], seq 1:187, ack 393, win 1027, options [nop,nop,TS val 1182007632 ecr 3797168361], length 186
19:45:23.772846 IP 192.168.2.1.46325 > 155.133.230.50.27025: Flags [.], ack 187, win 32406, options [nop,nop,TS val 3797168614 ecr 1182007632], length 0
19:45:23.819675 IP 155.133.230.50.27025 > 192.168.2.1.46325: Flags [P.], seq 187:470, ack 393, win 1027, options [nop,nop,TS val 1182007679 ecr 3797168614], length 283
19:45:23.819685 IP 192.168.2.1.46325 > 155.133.230.50.27025: Flags [.], ack 470, win 32406, options [nop,nop,TS val 3797168661 ecr 1182007679], length 0
19:45:25.743179 IP 192.168.2.1.46325 > 155.133.230.50.27025: Flags [P.], seq 393:507, ack 470, win 32406, options [nop,nop,TS val 3797170584 ecr 1182007679], length 114
19:45:25.796383 IP 155.133.230.50.27025 > 192.168.2.1.46325: Flags [.], ack 507, win 1027, options [nop,nop,TS val 1182009656 ecr 3797170584], length 0
19:45:26.522860 IP 155.133.230.50.27025 > 192.168.2.1.46325: Flags [P.], seq 470:638, ack 507, win 1027, options [nop,nop,TS val 1182010382 ecr 3797170584], length 168
19:45:26.522877 IP 192.168.2.1.46325 > 155.133.230.50.27025: Flags [.], ack 638, win 32406, options [nop,nop,TS val 3797171364 ecr 1182010382], length 0
^C
11 packets captured
15 packets received by filter
0 packets dropped by kernel
--------------------------------------------------------

From time to time some packets to Akamai servers (2.22.119.11, 104.81.127.75, 72.247.182.163) pop up via HTTPS, but not always. Can't find a pattern. Might be some things Steam uses for extra security.

According to https://www.protondb.com/app/306130 some people on Debian also had this problem.
Comment 4 Techwolf 2020-06-22 05:04:30 UTC
Oh wow, oh wow. Found this bug after not being able to update ESO online for a few days.

After much furpulling, I figure out the root cause.

What is causing the Launcher to "hang" at "Loading..." is a sig check of MinSpecDetectionInterop.dll failure. From the host.developer.log log.

06/16/2020 20:06:21 Loading interop library (C:\Program Files (x86)\Steam\steamapps\common\Zenimax Online\Launcher\MinSpecDetectionInterop.dll) (1.0.0.1)
06/16/2020 20:06:21 Certificate not trusted by trust provider
06/16/2020 20:06:21 Library validation (C:\Program Files (x86)\Steam\steamapps\common\Zenimax Online\Launcher\MinSpecDetectionInterop.dll) failed


I looked at the file on my backup ESO install on a win7 laptop. That file is signed with Zenimax Media Inc. with a CN of "thawte SHA256 Code signing CA" Serial number of 71a0b73695ddb1afc23b2b9a18ee54cb of the CA signer.
Comment 5 Techwolf 2020-06-22 05:49:34 UTC
I think I found the cause. Three files are missing in the upgrade.

/usr/share/ca-certificates/mozilla/thawte_Primary_Root_CA.crt
/usr/share/ca-certificates/mozilla/thawte_Primary_Root_CA_-_G2.crt
/usr/share/ca-certificates/mozilla/thawte_Primary_Root_CA_-_G3.crt

During the build of app-misc/ca-certificates-20200601.3.53::gentoo, I notice this:

Certificate "thawte Primary Root CA" blacklisted, ignoring.
Certificate "thawte Primary Root CA - G2" blacklisted, ignoring.
Certificate "thawte Primary Root CA - G3" blacklisted, ignoring.

The missing certs is what causing ESO online game Launcher/patcher failure to fully start.

Info at https://knowledge.digicert.com/generalinformation/INFO2172.html suggest that "thawte Primary Root CA" is needed for "thawte SHA256 Code signing CA".
Comment 6 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2020-06-22 18:01:45 UTC
(In reply to Techwolf from comment #5)
> I think I found the cause. Three files are missing in the upgrade.
> 
> /usr/share/ca-certificates/mozilla/thawte_Primary_Root_CA.crt
> /usr/share/ca-certificates/mozilla/thawte_Primary_Root_CA_-_G2.crt
> /usr/share/ca-certificates/mozilla/thawte_Primary_Root_CA_-_G3.crt
> 
> During the build of app-misc/ca-certificates-20200601.3.53::gentoo, I notice
> this:
> 
> Certificate "thawte Primary Root CA" blacklisted, ignoring.
> Certificate "thawte Primary Root CA - G2" blacklisted, ignoring.
> Certificate "thawte Primary Root CA - G3" blacklisted, ignoring.
> 
> The missing certs is what causing ESO online game Launcher/patcher failure
> to fully start.
> 
> Info at https://knowledge.digicert.com/generalinformation/INFO2172.html
> suggest that "thawte Primary Root CA" is needed for "thawte SHA256 Code
> signing CA".

Thanks for the extensive digging. I can tell you WHY those certs are gone now:

These are the Symantec related CAs that were removed because Symantec failed SSL audit requirements.
https://groups.google.com/forum/#!msg/mozilla.dev.security.policy/FLHRT79e3XE/90qkf8jsAQAJ

https://wiki.mozilla.org/CA:Symantec_Issues

So now I'm not sure what the best course of action here is. Games on steam are unlikely to get that file re-signed. The Official recommendations on Bethesda forums for windows are basically get that CA cert added back to the Windows Cert store by horrible hackery.

I don't know if there is a good way to override the certs for JUST one game either.
Comment 7 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2020-08-21 06:24:52 UTC
@TechWolf & others:
The old ca-certificate packages have been removed from the tree now.

If you still need specific now-removed certificates for Steam games, you should add those certificates to /usr/local/share/ca-certificates and then run update-ca-certificates.
Comment 8 Techwolf 2020-12-22 22:16:59 UTC
(In reply to Robin Johnson from comment #7)
> @TechWolf & others:
> The old ca-certificate packages have been removed from the tree now.
> 
> If you still need specific now-removed certificates for Steam games, you
> should add those certificates to /usr/local/share/ca-certificates and then
> run update-ca-certificates.

Just for clarification, in my case, copy /usr/share/ca-certificates/mozilla/thawte_Primary_Root_CA.crt and the two other files to said directory location. Update app-misc/ca-certificates package. Then run update-ca-certificates. Making sure I and others that find this bug though a search engine have a working workaround until its fixed upstream.
Comment 9 Tomasz Golinski 2020-12-22 22:57:53 UTC
For me, the game now works without any workarounds. However according to report on Valve's tracker, certificates-3.60 might break it again: https://github.com/ValveSoftware/Proton/issues/556#issuecomment-745458089