This one is very weird. While updating my ARM system, all my wget HTTPS downloads started failing. In short, this happens: $ wget -O /dev/null --debug -v https://gentoo.org Setting --verbose (verbose) to 1 DEBUG output created by Wget 1.21.2 on linux-gnueabihf. Reading HSTS entries from /home/chewi/.wget-hsts --2021-12-02 09:19:40-- https://gentoo.org/ SSL_INIT Certificates loaded: 131 Resolving gentoo.org... 89.16.167.134, 2001:41c8:0:936::139, 2001:41c8:0:936::136 Caching gentoo.org => 89.16.167.134 2001:41c8:0:936::139 2001:41c8:0:936::136 Connecting to gentoo.org|89.16.167.134|:443... connected. Created socket 4. Releasing 0x012d78f8 (new refcount 1). The certificate has not yet been activated That last line refers to the certificate start date. I added some extra information to that message and found that it was reporting start times like 8566854470176633495 instead of a more current time like 1638437181. While writing this report, I decided to give the older 1.21.1 a try. I didn't expect it to work, but it did! I will dig into this and report back. For the record though, gnutls-cli, wget + OpenSSL, and curl + GnuTLS all work fine. It's just wget + GnuTLS that's broken. ---- Portage 3.0.28 (python 3.9.9-final-0, default/linux/arm/17.0/armv7a, gcc-11.2.0, glibc-2.34-r2, 5.15.5-00011-g094ddf25878e armv7l) ================================================================= System uname: Linux-5.15.5-00011-g094ddf25878e-armv7l-ARMv7_Processor_rev_10_-v7l-with-glibc2.34 KiB Mem: 2062492 total, 862772 free KiB Swap: 2097148 total, 2097148 free Head commit of repository gentoo: 0fb1858b551ad69127463f27fb4b30a649b0500c sh bash 5.1_p12 ld GNU ld (Gentoo 2.36.1 p3) 2.36.1 distcc 3.4 armv7a-unknown-linux-gnueabihf [disabled] app-shells/bash: 5.1_p12::gentoo dev-java/java-config: 2.3.1::gentoo dev-lang/perl: 5.34.0-r5::gentoo dev-lang/python: 3.9.9::gentoo, 3.10.0_p1::gentoo dev-util/cmake: 3.22.0::gentoo sys-apps/baselayout: 2.8::gentoo sys-apps/openrc: 0.44.8::gentoo sys-apps/sandbox: 2.29::gentoo sys-devel/autoconf: 2.69-r5::gentoo, 2.71-r1::gentoo sys-devel/automake: 1.13.4-r2::gentoo, 1.15.1-r2::gentoo, 1.16.5::gentoo sys-devel/binutils: 2.36.1-r1::gentoo, 2.37_p1-r1::gentoo sys-devel/gcc: 11.2.0::gentoo sys-devel/gcc-config: 2.5-r1::gentoo sys-devel/libtool: 2.4.6-r6::gentoo sys-devel/make: 4.3::gentoo sys-kernel/linux-headers: 5.15::gentoo (virtual/os-headers) sys-libs/glibc: 2.34-r2::gentoo Repositories: gentoo location: /usr/portage sync-type: git sync-uri: git+ssh://git@git.gentoo.org/repo/gentoo.git priority: -1000 vivaldi location: /home/chewi/Projects/vivaldi-overlay masters: gentoo priority: 1001 ACCEPT_KEYWORDS="arm ~arm" ACCEPT_LICENSE="*" CBUILD="armv7a-unknown-linux-gnueabihf" CFLAGS="-mcpu=cortex-a9 -mfpu=neon -O3 -pipe" CHOST="armv7a-unknown-linux-gnueabihf" CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php7.4/ext-active/ /etc/php/apache2-php8.0/ext-active/ /etc/php/cgi-php7.4/ext-active/ /etc/php/cgi-php8.0/ext-active/ /etc/php/cli-php7.4/ext-active/ /etc/php/cli-php8.0/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-mcpu=cortex-a9 -mfpu=neon -O3 -pipe" DISTDIR="/usr/portage/distfiles" EMERGE_DEFAULT_OPTS="--ask-enter-invalid --quiet-build=n --nospinner" ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR" FCFLAGS="-O2 -pipe -march=armv7-a" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms sign strict strict-keepdir unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe -march=armv7-a" GENTOO_MIRRORS="https://mirror.bytemark.co.uk/gentoo https://gentoo.osuosl.org" LANG="en_GB.UTF-8" LC_ALL="en_GB.UTF-8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LINGUAS="en en_GB" MAKEOPTS="-j4" PKGDIR="/var/cache/binpkgs" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" USE="S3TC X509 a52 aac aalib aften alaw amr arm armvfp asf async audiofile autoipd barcode bash-completion blender-game bluetooth bluray bmp bogofilter boost branding bzip2 cairo caps cegui chardet cleartype cli community console crypt cscope css ctype curl dbus device-mapper dhcp dri dts dv dvb dvd dvdnav dvdr egl elogind emf emoticon enca encode exceptions exif faad fam fat fbcon fbdev fftw filter firefox flac fluidsynth fmod fontconfig ftp fuse g3dvl g722 g729 gallium gbm gdbm gdu gentoo-dev geos gif git gles gles1 gles2 glut gmp gnutls grammar gsl gsm gstreamer gudev hash hddtemp headless-awt hires-icons hog http http2 hwdb icon iconv icu id3 id3tag ilbc image imagemagick imlib ipv6 ithreads jabber joystick jpeg keymap kms kvm ladspa libffi libglvnd libkms libnotify libvisual lights live lj lm_sensors lvm lzo m17n-lib mad maildir matroska mbox mbrola md5sum midi minizip mmap mod modplug mp3 mp4 mpeg mplayer music mvl ncurses neon network nfs nfsidmap nfsv3 nfsv4 nfsv41 nocd nptl ntfs ntfsprogs offensive ogg openal opengl openmp openssl openvg openxml opus pam pcre pg-intdatetime pipewire png pnm posix ppds pulseaudio qmax qt3support quicktime rar readline realtime rtc rtsp scanner scrobbler sdl sdl-image sdl-sound sdlaudio seccomp secure-delete sha512 simplexml skins smp sndfile soap sockets sound soundex sounds soundtouch speex spell split-usr ssl startup-notification stemmer stream sysvipc taglib textures tftp tga theora thesaurus threads thumbnail thunar timidity tordns transparent-proxy truetype udev uk_rt ulaw unicode unzip usb userlocales vcd vhosts videos vispatch vnc vorbis vpx wav wayland web webgl webkit webm webp wifi win32 wma x264 xattr xml xmms2 xosd xsl xvid zip zlib" ADA_TARGET="gnat_2020" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_ARM="edsp neon thumb vfp vfpv3 vfp-d32 v4 v5 v6 v7 thumb2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="evdev" KERNEL="linux" L10N="en-GB" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" NGINX_MODULES_HTTP="access auth_basic autoindex fastcgi geoip gzip proxy rewrite ssi" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-3 php7-4" POSTGRES_TARGETS="postgres12 postgres13" PYTHON_SINGLE_TARGET="python3_9" PYTHON_TARGETS="python3_9" QEMU_SOFTMMU_TARGETS="arm" QEMU_USER_TARGETS="arm" RUBY_TARGETS="ruby26 ruby27" USERLAND="GNU" VIDEO_CARDS="kmsro imx vivante" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, RUSTFLAGS
The code which is triggering is https://github.com/mirror/wget/blob/d5588ac2216a6541a7362c0605828e108cfb4347/src/gnutls.c#L1086-L1090 (In reply to James Le Cuirot from comment #0) > That last line refers to the certificate start date. I added some extra > information to that message and found that it was reporting start times like > 8566854470176633495 instead of a more current time like 1638437181. wget switched to gnulib's utime.h (https://github.com/mirror/wget/commit/7d9ed223fca5ce163b0de462b97fec33a9769518) in 1.21.2. 1.21.2 is using gnulib from 2021-09-07. I wonder if it is failing because of the year2038 support which was added in July for glibc-2.34 to gnulib.
(In reply to Thomas Deutschmann from comment #1) > The code which is triggering is > https://github.com/mirror/wget/blob/d5588ac2216a6541a7362c0605828e108cfb4347/ > src/gnutls.c#L1086-L1090 > > (In reply to James Le Cuirot from comment #0) > > That last line refers to the certificate start date. I added some extra > > information to that message and found that it was reporting start times like > > 8566854470176633495 instead of a more current time like 1638437181. > > wget switched to gnulib's utime.h > (https://github.com/mirror/wget/commit/ > 7d9ed223fca5ce163b0de462b97fec33a9769518) in 1.21.2. > > 1.21.2 is using gnulib from 2021-09-07. I wonder if it is failing because of > the year2038 support which was added in July for glibc-2.34 to gnulib. I haven't had a chance to look at it yet, but that sounds highly plausible. 2038 support had already crossed my mind.
Hmm, reverting 7d9ed223fca5ce163b0de462b97fec33a9769518 actually didn't help.
I've bisected the problem to 5a79362a56a69c97f9cc1d598f3e933b5ec640fc, which was the gnulib update on 2021-09-07 that you mentioned. I'm now bisecting gnulib while keeping wget itself on that commit.
As you would expect, the bad gnulib commit is dc09dc0888485698a8e74205b9df43159aef0f61, the one that added year2038 support. wget has a --disable-year2038 configure option, and it works if you pass that. Curiously, Fedora's wget is built against GnuTLS, so I tried Fedora 35 in a chroot and it works there. This could be Gentoo-specific?
Oh haha, their RPM spec file has --disable-year2038. It doesn't say why they added that. Maybe they hit this problem too.
Does gnutls use time_t in its API? It might behave quite strangely if wget starts passing 8-byte time_t values where gnutls expects a 4-byte value. I suspect we would need to coordinate enabling 64-bit time_t across any packages that call such a library.
(In reply to Mike Gilbert from comment #7) > Does gnutls use time_t in its API? It might behave quite strangely if wget > starts passing 8-byte time_t values where gnutls expects a 4-byte value. > > I suspect we would need to coordinate enabling 64-bit time_t across any > packages that call such a library. From what I've seen, gnutls should be year 2038 aware, but I need to take a closer look. It also works on 32-bit x86. I have read that 32-bit ARM is a special case, but I don't know why. https://www.mail-archive.com/bug-gnulib@gnu.org/msg41083.html
One of the last things reported by GnuTLS' configure script is this: checking size of time_t... 4 I get that on arm, x86, and amd64 though! I still don't understand why only arm is broken. I have reached out to the Red Hat maintainer of wget in Fedora. Maybe he knows more about it.
(In reply to James Le Cuirot from comment #9) > One of the last things reported by GnuTLS' configure script is this: > > checking size of time_t... 4 > > I get that on arm, x86, and amd64 though! Your result on amd64 seems like an error. time_t has always been 8 bytes on amd64, and I have just verified that locally. I would guess that you misread something. time_t is 4 bytes on x86, unless you define _TIME_BITS=64 before including time.h. > I still don't understand why only arm is broken. A guess: maybe this has something to do with the differing ways in which ARM and x86 pass arguments to/from functions. x86 always uses the stack, whereas ARM might use registers?
I can reproduce the problem on x86 with glibc-2.34, wget-1.21.2, and gnutls-3.7.2. > i686 ~ # wget https://icanhazip.com/ > --2021-12-13 00:55:28-- https://icanhazip.com/ > SSL_INIT > Resolving icanhazip.com... 104.18.115.97, 104.18.114.97, 2606:4700::6812:7261, ... > Connecting to icanhazip.com|104.18.115.97|:443... connected. > The certificate has not yet been activated
Hmm, rebuilding net-libs/gnutls on x86 resolved the issue for me.
(In reply to Mike Gilbert from comment #10) > Your result on amd64 seems like an error. time_t has always been 8 bytes on > amd64, and I have just verified that locally. I would guess that you misread > something. Before checking, I thought to myself "make sure you don't look at multilib". Guess what I did. ;) > Hmm, rebuilding net-libs/gnutls on x86 resolved the issue for me. Interesting! I had built gnutls on x86 after updating glibc, so that's consistent. However, rebuilding doesn't help on arm. I'd already tried it, and have just tried it again to make sure. It certainly isn't the first thing I've had to rebuild after glibc 2.34. I saw breakage with busybox on m68k and openrc on arm. busybox was statically linked, which probably had something to do with it. I don't know why openrc broke though.
I'll try to do some debugging to see if I can figure out why rebuilding gnutls makes this magically work on x86. That really doesn't make any sense to me. My working theory (yet to be confirmed): gnutls_x509_crt_get_activation_time returns a time_t. On x86, time_t is a 4-byte integer by default. It is returned to the caller in the eax register. When wget is compiled with _TIME_BITS=64, it treats time_t as an 8-byte integer (type long long). It expects gnutls_x509_crt_get_activation_time to return 2 4-byte values in the eax and edx registers. These are the low and high words of the 8-byte return value. Before the gnutls recompile, gnutls_x509_crt_get_activation_time returns a valid value in eax, and junk data in edx. Due to luck or some subtle behavior change, the recompiled copy of gnutls returns 0 in edx instead of junk data.
I tried adding the gnulib year2038 module to GnuTLS, and that fixes it! It looks like git master already has that but haven't tried that yet. That makes me worry about other consumers of GnuTLS though. I'll test things the other way round.
Any mismatch in the size of time_t between gnutls and its reverse deps is going to be problematic and cause weird failures at run time. I would suggest we disable 64-bit time_t for all packages until we can identify all affected libraries and their reverse dependencies.
(In reply to James Le Cuirot from comment #9) > One of the last things reported by GnuTLS' configure script is this: > > checking size of time_t... 4 > > I get that on arm, x86, and amd64 though! I still don't understand why only > arm is broken. x86 is also broken. I've hit this bug on ~x86 box.
(In reply to Andrew Savchenko from comment #17) > x86 is also broken. I've hit this bug on ~x86 box. Chewi did clarify he made a mistake if you read the rest and floppym already noted it's broken on x86 ;) You're free to come to #gentoo-toolchain and help us with the time64 migration planning: https://wiki.gentoo.org/wiki/Project:Toolchain/time64_migration
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=89d7cf613dc997bcaea196b22295328192fc9ef8 commit 89d7cf613dc997bcaea196b22295328192fc9ef8 Author: Mike Gilbert <floppym@gentoo.org> AuthorDate: 2021-12-17 14:36:27 +0000 Commit: Mike Gilbert <floppym@gentoo.org> CommitDate: 2021-12-20 19:00:07 +0000 profiles/default/linux: set gl_cv_type_time_t_bits_macro=no This is intended to prevent packages from automatically switching to 64-bit time_t on 32-bit ABIs. Making this switch in an uncontrolled manner will lead to inconsistent library ABIs that fail at runtime. At a later time, we will take steps to enable 64-bit time_t distro-wide. https://wiki.gentoo.org/wiki/Project:Toolchain/time64_migration Bug: https://bugs.gentoo.org/828001 Signed-off-by: Mike Gilbert <floppym@gentoo.org> profiles/default/linux/make.defaults | 5 +++++ 1 file changed, 5 insertions(+)
Is this fixed now then?
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=00f2388f06a9681e4050be48aa7caa1bd1c1b861 commit 00f2388f06a9681e4050be48aa7caa1bd1c1b861 Author: Mike Gilbert <floppym@gentoo.org> AuthorDate: 2022-02-22 17:29:03 +0000 Commit: Mike Gilbert <floppym@gentoo.org> CommitDate: 2022-02-22 17:29:03 +0000 profiles/default/linux: set enable_year2038="no" This bypasses the 64-bit time_t configure logic entirely, and prevents configure failures on 32-bit systems where /usr/bin/touch supports 64-bit timestamps. Bug: https://bugs.gentoo.org/828001 Signed-off-by: Mike Gilbert <floppym@gentoo.org> profiles/default/linux/make.defaults | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=f6caba28e1a825acc76709c406c3b987253bbcc7 commit f6caba28e1a825acc76709c406c3b987253bbcc7 Author: Andreas K. Hüttel <dilfridge@gentoo.org> AuthorDate: 2023-10-10 20:16:41 +0000 Commit: Andreas K. Hüttel <dilfridge@gentoo.org> CommitDate: 2023-10-10 20:17:35 +0000 riscv32: force 64bit time on (there is no other variant here) Bug: https://bugs.gentoo.org/828001 Signed-off-by: Andreas K. Hüttel <dilfridge@gentoo.org> profiles/arch/riscv/rv32imac/ilp32/make.defaults | 9 +++++++++ profiles/arch/riscv/rv32imac/ilp32d/make.defaults | 9 +++++++++ 2 files changed, 18 insertions(+)
