Summary: | dev-db/mariadb-10.6.5-r1 cannot read database on linux-5.16+ with BTRFS ([ERROR] InnoDB: Database page corruption on disk or a failed read of file './ibdata1') (was: cannot initialise a clean installation (emerge --config fails) | ||
---|---|---|---|
Product: | Gentoo Linux | Reporter: | Mike Benson <mike> |
Component: | Current packages | Assignee: | Gentoo Linux MySQL bugs team <mysql-bugs> |
Status: | RESOLVED TEST-REQUEST | ||
Severity: | critical | CC: | ao, dennis.lissov, dschridde+gentoobugs, hydrapolic, kernel, sam, whissi |
Priority: | High | ||
Version: | unspecified | ||
Hardware: | All | ||
OS: | Linux | ||
Whiteboard: | |||
Package list: | Runtime testing required: | --- | |
Attachments: |
emerge --info
install_db.log mysqld.err mariadb-10.6.5-r2.ebuild mysqld.err Error log from mariadb server (mysqld) package update history since 2022-01-31 kernel config diff gentoo-kernel-5.16.5 config (complete) |
Created attachment 763546 [details]
install_db.log
Created attachment 763547 [details]
mysqld.err
Created attachment 763548 [details]
mariadb-10.6.5-r2.ebuild
Modified ebuild with fixes for plugin directory search in mariadb-install-db and mysqld. The first and second problems are fixed, but not the third (./ibdata1 corrupt)
Created attachment 763549 [details]
mysqld.err
Error file from mariadb server during pkg_config()
Created attachment 763550 [details]
Error log from mariadb server (mysqld)
Latest run with updated ebuild
Please don't manually add arches to cc list. Thanks I can confirm the corrupt ibdata1 file for both existing installations / databases as well as newly created ones. I also experience this with Akonadi, namely kde-apps/akonadi-21.12.1. Created attachment 764296 [details]
package update history since 2022-01-31
Akonadi worked fine before updating the attached packages (and possibly still some time in between 2022-01-31 and now, 2022-02-03).
Created attachment 764297 [details]
kernel config diff
Starting Linux 5.15.18 instead of 5.16.5 resolves the problem for me. The kernel config changed a little bit between the versions. See attached diff.
Created attachment 764298 [details]
gentoo-kernel-5.16.5 config (complete)
Persists with Linux 5.16.6 and dev-db/mariadb-10.6.5-r1. Portage 3.0.30 (python 3.9.10-final-0, default/linux/amd64/17.1/desktop/plasma/systemd, gcc-11.2.1, glibc-2.34-r8, 5.16.6-gentoo-dist x86_64) ================================================================= System Settings ================================================================= System uname: Linux-5.16.6-gentoo-dist-x86_64-AMD_Ryzen_5_2400G_with_Radeon_Vega_Graphics-with-glibc2.34 KiB Mem: 14186520 total, 1781456 free KiB Swap: 4194300 total, 4194300 free Timestamp of repository gentoo: Mon, 07 Feb 2022 12:15:01 +0000 Head commit of repository gentoo: 80f127e2d619a214b198f3958421bc87b3582a85 Timestamp of repository haskell: Sun, 06 Feb 2022 16:51:19 +0000 Head commit of repository haskell: 32b4e4858bd5239d8b6691788716986dd75e9fd1 Head commit of repository local: bcfadff1cc09af836647ac57ae1205efde0e7ba5 sh bash 5.1_p16 ld GNU ld (Gentoo 2.37_p1 p2) 2.37 ccache version 4.5.1 [disabled] app-misc/pax-utils: 1.3.3::gentoo app-shells/bash: 5.1_p16::gentoo dev-java/java-config: 2.3.1::gentoo dev-lang/perl: 5.34.0-r6::gentoo dev-lang/python: 2.7.18_p13::gentoo, 3.9.10-r1::gentoo, 3.10.2-r1::gentoo dev-lang/rust: 1.58.1::gentoo dev-util/ccache: 4.5.1::gentoo dev-util/cmake: 3.22.2::gentoo dev-util/meson: 0.60.3::gentoo sys-apps/baselayout: 2.8::gentoo sys-apps/sandbox: 2.29::gentoo sys-apps/systemd: 250.3::gentoo sys-devel/autoconf: 2.13-r1::gentoo, 2.71-r1::gentoo sys-devel/automake: 1.16.5::gentoo sys-devel/binutils: 2.37_p1-r2::gentoo sys-devel/binutils-config: 5.4::gentoo sys-devel/clang: 13.0.1::gentoo sys-devel/gcc: 11.2.1_p20220115::gentoo sys-devel/gcc-config: 2.5-r1::gentoo sys-devel/libtool: 2.4.6-r6::gentoo sys-devel/lld: 13.0.1::gentoo sys-devel/llvm: 13.0.1::gentoo sys-devel/make: 4.3::gentoo sys-kernel/linux-headers: 5.16::gentoo (virtual/os-headers) sys-libs/glibc: 2.34-r8::gentoo Repositories: gentoo location: /var/cache/portage/gentoo sync-type: rsync sync-uri: rsync://rsync.de.gentoo.org/gentoo-portage priority: -1000 sync-rsync-extra-opts: --new-compress sync-rsync-verify-metamanifest: yes sync-rsync-verify-jobs: 1 sync-rsync-verify-max-age: 24 haskell location: /var/db/repos/haskell sync-type: git sync-uri: https://github.com/gentoo-mirror/haskell.git masters: gentoo crossdev location: /var/cache/portage/crossdev masters: gentoo priority: 100 local location: /var/cache/portage/local sync-type: git sync-uri: https://github.com/devurandom/gentoo-overlay.git masters: gentoo priority: 1000 ACCEPT_KEYWORDS="amd64 ~amd64" ACCEPT_LICENSE="@FREE" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-pipe -O2 -march=znver1" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/maven-bin-3.8/conf" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c" CXXFLAGS="-pipe -O2 -march=znver1" DISTDIR="/var/cache/portage/distfiles" EMERGE_DEFAULT_OPTS="--nospinner" ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs binpkg-multi-instance buildpkg-live buildsyspkg cgroup compressdebug config-protect-if-modified distlocks ebuild-locks fakeroot fixlafiles ipc-sandbox merge-sync mount-sandbox multilib-strict network-sandbox news parallel-fetch parallel-install pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://linux.rz.ruhr-uni-bochum.de/download/gentoo-mirror/ http://ftp.fau.de/gentoo http://ftp.halifax.rwth-aachen.de/gentoo/ http://ftp-stud.hs-esslingen.de/pub/Mirrors/gentoo/ http://distfiles.gentoo.org" LANG="en_US.UTF-8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j6 -l4" PKGDIR="/var/cache/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_EXTRA_OPTS="--new-compress" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/tmp" SHELL="/usr/bin/bash" USE="7z 7zip X a52 aac aacplus aacs acl acpi activities aio alsa amd64 amr appindicator appstream aptx archive audit avahi avif ayatana bdplus blake2 blas bluetooth bluray bpf branding brotli bs2b btrfs bzip2 cairo caps cdda cddb cdio cdr celt chromaprint cjk cli clipboard color-management colord conntrack crypt cups curl d3d9 dav1d dbus declarative device-mapper dirac djvu dmabuf dri drm dts dvb dvd dvdr ed25519 editorconfig egl elf emoji encode epub evdev exif faudio fax fdk ffmpeg fftw filecaps firefox firewalld fish-completion fits flac fontconfig fontforge fortran fribidi gamepad gbm gdal gdbm geoclue geolocation gif git gles2 gles3 gmp gnome-online-accounts gnupg google googledrive gpg gps graphicsmagick graphite gstreamer gtk gtk3 gui gzip harfbuzz hdf5 heif http2 hugepages ibus iconv icu id3tag idn imlib inotify introspection ipv6 iwd jemalloc jpeg jpeg2k json kde kms kwallet lapack latex lcms ldac libatomic libglvnd libidn2 libinput libnotify libproxy libsecret libsoxr libtirpc libvirt lm-sensors lrz lto lv2 lvm lz4 lzma lzo mad magic man mariadb markdown mbim mercurial mjpeg mng mobi modemmanager modplug mp3 mp4 mpeg mplayer mpris mtp multilib mysql ncurses netlink networkmanager nftables nls nptl numa office ofx ogg openal opencl opencv openexr opengl openh264 openmax openmp opus otf pam pango pcap pcre pcre2 pdf pgo pipewire pixman pkcs11 pkcs7 plasma pm-utils png policykit postscript ppds prison pulseaudio python qml qrcode qt5 rav1e raw rdp readline rtmp rubberband samba sasl scanner schroedinger screencast sctp sdl sdl2 seatd seccomp semantic-desktop share smartcard snappy sparse speech speex spell spice srt ssl startup-notification steamruntime stemmer svg systemd tbb teamd telemetry telepathy tga theora threads thunderbolt tiff timezone tmux tpm truetype udev udisks uinput unicode unwind upnp upnp-av upower usb utempter v4l v4l2 vaapi vamp vdpau vkd3d vorbis vpx vtk vulkan wasm wavpack wayland webchannel webengine webp widgets wireguard wmf woff2 wps x264 x265 xattr xcb xinerama xkb xml xmp xrandr xscreensaver xv xvid xwayland xxhash xz yaml zeroconf zeromq zimg zlib zstd" ABI_X86="64" ADA_TARGET="gnat_2020" ALSA_CARDS="hda-intel" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt rdrand sha sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="efi-64" INPUT_DEVICES="joystick libinput" KERNEL="linux" L10N="de de-DE en en-GB ar fa tr ja ko zh zh-CN zh-TW" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="nlpsolver scripting-javascript wiki-publisher" LLVM_TARGETS="AMDGPU BPF RISCV WebAssembly" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1 lua5-4 luajit" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-4 php8-0" POSTGRES_TARGETS="postgres12 postgres13" PYTHON_SINGLE_TARGET="python3_9" PYTHON_TARGETS="python3_9 pypy pypy3" QEMU_SOFTMMU_TARGETS="riscv32 riscv64 x86_64" QEMU_USER_TARGETS="riscv32 riscv64" RUBY_TARGETS="ruby26 ruby27" USERLAND="GNU" VIDEO_CARDS="amdgpu virgl" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LC_ALL, LD, LEX, LFLAGS, LIBTOOL, LINGUAS, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, RANLIB, READELF, RUSTFLAGS, SIZE, STRINGS, STRIP, YACC, YFLAGS ================================================================= Package Settings ================================================================= sys-kernel/gentoo-kernel-5.15.18::gentoo was built with the following: USE="initramfs savedconfig -debug -hardened -test" ABI_X86="(64)" FEATURES="parallel-install fixlafiles userfetch usersandbox distlocks ebuild-locks binpkg-logs compressdebug strict multilib-strict sfperms preserve-libs unmerge-logs ipc-sandbox unmerge-orphans assume-digests binpkg-docompress network-sandbox sandbox unknown-features-warn cgroup buildsyspkg news fakeroot binpkg-multi-instance pid-sandbox config-protect-if-modified mount-sandbox qa-unresolved-soname-deps buildpkg-live merge-sync xattr userpriv protect-owned parallel-fetch binpkg-dostrip usersync" sys-kernel/gentoo-kernel-5.16.6::gentoo was built with the following: USE="initramfs savedconfig -debug -hardened -test" ABI_X86="(64)" FEATURES="merge-sync preserve-libs fixlafiles qa-unresolved-soname-deps mount-sandbox binpkg-docompress xattr binpkg-dostrip buildpkg-live userfetch unknown-features-warn buildsyspkg ebuild-locks protect-owned assume-digests parallel-install sandbox strict config-protect-if-modified binpkg-multi-instance fakeroot compressdebug unmerge-orphans sfperms usersync multilib-strict parallel-fetch userpriv ipc-sandbox usersandbox pid-sandbox distlocks network-sandbox news binpkg-logs unmerge-logs cgroup" dev-db/mariadb-10.6.5-r1::gentoo was built with the following: USE="backup innodb-lz4 innodb-lzo innodb-snappy jemalloc numa pam perl server systemd -bindist -columnstore -cracklib -debug -extraengine -galera -jdbc -kerberos -latin1 -mroonga -odbc -oqgraph -profiling -rocksdb -s3 (-selinux) -sphinx -sst-mariabackup -sst-rsync -static -systemtap -tcmalloc -test -xml -yassl" ABI_X86="(64)" CFLAGS="-pipe -O2 -march=znver1 -fno-strict-aliasing" CXXFLAGS="-pipe -O2 -march=znver1 -felide-constructors -fno-strict-aliasing" FEATURES="binpkg-docompress config-protect-if-modified cgroup unmerge-logs network-sandbox qa-unresolved-soname-deps unmerge-orphans usersync multilib-strict userpriv buildsyspkg strict unknown-features-warn news assume-digests binpkg-dostrip sfperms userfetch fakeroot protect-owned sandbox preserve-libs xattr merge-sync distlocks fixlafiles compressdebug binpkg-multi-instance ipc-sandbox parallel-fetch parallel-install ebuild-locks pid-sandbox binpkg-logs usersandbox mount-sandbox buildpkg-live" Persists with Linux 5.16.9. I found a workaround, though. I put `innodb_use_native_aio=0` in the `[mysqld]` section of `$HOME/.local/share/akonadi/mysql.conf` and now Akonadi / MariaDB starts. I got this idea when reading the MariaDB 10.6.5 release notes: https://mariadb.com/kb/en/mariadb-1065-release-notes/#innodb (In reply to Dennis Schridde from comment #13) > Persists with Linux 5.16.9. > > I found a workaround, though. I put `innodb_use_native_aio=0` in the > `[mysqld]` section of `$HOME/.local/share/akonadi/mysql.conf` and now > Akonadi / MariaDB starts. I got this idea when reading the MariaDB 10.6.5 > release notes: https://mariadb.com/kb/en/mariadb-1065-release-notes/#innodb Confirmed both the behavior and the workaround, thanks a lot! Linux: 5.16.11 mariadb 10.6.5-r1 (In reply to Fabio Coatti from comment #14) > (In reply to Dennis Schridde from comment #13) > > Persists with Linux 5.16.9. > > > > I found a workaround, though. I put `innodb_use_native_aio=0` in the > > `[mysqld]` section of `$HOME/.local/share/akonadi/mysql.conf` and now > > Akonadi / MariaDB starts. I got this idea when reading the MariaDB 10.6.5 > > release notes: https://mariadb.com/kb/en/mariadb-1065-release-notes/#innodb > > Confirmed both the behavior and the workaround, thanks a lot! > Linux: 5.16.11 > mariadb 10.6.5-r1 Thank you for confirming the workaround. I am glad that it helped you! You are using linux-5.16+ and BTRFS, right? (In reply to Thomas Deutschmann from comment #16) > You are using linux-5.16+ and BTRFS, right? Yes. Exact kernel version is in the comments above. All my file systems are on Btrfs. (In reply to Thomas Deutschmann from comment #16) > You are using linux-5.16+ and BTRFS, right? 5.16.11-gentoo-x86_64, and yes, I am using BTRFS. Dennis' workaround is also working for me. (In reply to Dennis Schridde from comment #13) > Persists with Linux 5.16.9. > > I found a workaround, though. I put `innodb_use_native_aio=0` in the > `[mysqld]` section of `$HOME/.local/share/akonadi/mysql.conf` and now > Akonadi / MariaDB starts. I got this idea when reading the MariaDB 10.6.5 > release notes: https://mariadb.com/kb/en/mariadb-1065-release-notes/#innodb P.S. It appears that this needs to be re-done after each Akonadi update. *** Bug 834532 has been marked as a duplicate of this bug. *** From bug 834532: (In reply to Thomas Deutschmann from comment #0) > Folks, > > Upon raising https://jira.mariadb.org/browse/MDEV-27900, and reraising > it to the btrfs kernel developers, Filipe Manana (Suse) came up with > this patch. > > https://lore.kernel.org/linux-btrfs/ > CABVffENfbsC6HjGbskRZGR2NvxbnQi17gAuW65eOM+QRzsr8Bg@mail.gmail.com/T/#t > > As a regression to 51bd9563b6783d, this affects all 5.16 kernels. > > Bug report will contain an error log like: > > 2022-02-21 14:59:31 0 [ERROR] InnoDB: Database page corruption on disk > or a failed read of file './ibdata1' page [page id: space=0, page > number=9]. You may have to recover from a backup. > 2022-02-21 14:59:31 0 [Note] InnoDB: Page dump in ascii and hex (16384 > bytes): > len 16384; hex > 0000000000000009ffffffffffffffff0000000000009c2045bf0000000000000000000000000 > 002012022-02-21 > 14:59:31 0 [ERROR] InnoDB: Database page corruption on disk or a > failed read of file './ibdata1' page [page id: space=0, page > number=243]. You may have to recover from a backup. > 710009002022-02-21 14:59:31 0 [Note] InnoDB: Page dump in ascii and > hex (16384 bytes): > > This has occurred exclusively on btrfs so far, however there are > suggestions on the linux-fsdevel list that gfs2 can also experience > the same. > > It's important to note, the users' data isn't corrupt, just MariaDB's > read of it. Until a kernel update is prepared they can work around the > bug with innodb_use_native_aio=0 or innodb_flush_method=fsync as > MariaDB configuration items. > > > In the meantime, MariaDB upstream is working on MDEV-27593 to handle InnoDB > AIO errors with message rather than assertion, to better validate and handle > error code, and returned sizes apparently. Depends on: https://bugs.gentoo.org/833760 Bug #833760 (dev-db/mariadb: version bumps) blocks this, because https://jira.mariadb.org/browse/MDEV-27900 was fixed in MariaDB 10.5.16, 10.6.8, 10.7.4, 10.8.3, but these versions are not yet in Gentoo. Please adjust the "depends on" field of this bug. (In reply to Dennis Schridde from comment #22) > Depends on: https://bugs.gentoo.org/833760 > > Bug #833760 (dev-db/mariadb: version bumps) blocks this, because > https://jira.mariadb.org/browse/MDEV-27900 was fixed in MariaDB 10.5.16, > 10.6.8, 10.7.4, 10.8.3, but these versions are not yet in Gentoo. Please > adjust the "depends on" field of this bug. Sorry, I just noticed these versions are not yet released. Nevertheless a backport would probably be easier, if we had the latest released versions available in Gentoo. For me Akonadi's mysql crashes during mail check even with innodb_use_native_aio=0. Is this to be expected? Comment 21 says users' data isn't corrupted, only the read fails. Is it to be expected? Should I delete & build up the db from scratch? I have Linux 5.17.7, mariadb 10.6.5-r1 and btrfs volume as well. Fixed versions in tree, please test. |
Created attachment 763545 [details] emerge --info I've been experiencing multiple cases where mariadb has crashed claiming corruption in the mysql database. (I run KDE, and akonadi using mariadb as the database server, so mariadb crashing is a major issue for me). I attempted to repair my problems by dumping my databases with mysqldump, and reinitializing the mysql database by running emerge --config =dev-db/mariadb-10.6.5-r1. It fails. Every time. First problem: mariadb-install-db does not set plugin_dir correctly. The now required authentication plugins are located in lib64/mariadb/plugin, which is not searched. Second problem: The subsequent invocation of mysqld in pkg_config() does not set plugin_dir. Third problem: mysqld crashes complaining that the ./ibdata1 file (created by mariadb-install-db above) is corrupt.