We have deployed a small test cluster consisting of three nodes. Each node is running a mon/mgr and two osds (Samsung PM983 3,84TB NVMe split into two partitions), so six osds in total. The configuration is almost unchanged compared to the defaults, we only set several debugging options to 0/0. After upgrading to Octopus (using sys-cluster/ceph-15.2.3 initially), we saw massive performance degradations compared to Nautilus (using sys-cluster/ceph-14.2.9) on the same hardware. After hours of looking into the issue, we finally figured out, that the problem is caused by the Ebuilds not setting CMAKE_BUILD_TYPE=RelWithDebInfo when building Ceph. The README.md in the Ceph source on https://github.com/ceph/ceph explicitly says: 'do_cmake.sh now defaults to creating a debug build of ceph that can be up to 5x slower with some workloads. Please pass "-DCMAKE_BUILD_TYPE=RelWithDebInfo" to do_cmake.sh to create a non-debug release.' The ebuilds are, to our understanding, not using do_cmake.sh, but setting CMAKE_BUILD_TYPE accordingly seems to be necessary nevertheless. After patching the Ebuild for sys-cluster/ceph-15.2.4 and rebuilding, we are now getting the same results as under Centos 8 using the official RPMs for Ceph 15.2.3, which we set up temporarily on the same hardware in order to narrow down the cause of the problem. Here are some test results with sys-cluster/ceph-15.2.4 and nine clients against one rbd volume each using fio. (Test | IOPS without setting CMAKE_BUILD_TYPE | IOPS with CMAKE_BUILD_TYPE=RelWithDebInfo) 4k/random/iodepth=32/read | 98000 | 162000 4k/random/iodepth=32/write | 15800 | 55000 4k/sequential/iodepth=32/read | 60900 | 60900 4k/sequential/iodepth=32/write | 82300 | 130800 We also retested sys-cluster/ceph-14.2.10 built with CMAKE_BUILD_TYPE=RelWithDebInfo and compared the results to a build using the unpatched Ebuild, but interestingly at least in our setup we have been seeing hardly any difference, so not setting CMAKE_BUILD_TYPE=RelWithDebInfo seems to affect the performance of Octopus in particular. Reproducible: Always Portage 2.3.103 (python 3.6.11-final-0, default/linux/amd64/17.1, gcc-9.3.0, glibc-2.30-r8, 5.4.48-gentoo x86_64) ================================================================= System uname: Linux-5.4.48-gentoo-x86_64-Intel-R-_Xeon-R-_CPU_E5-2690_v3_@_2.60GHz-with-gentoo-2.6 KiB Mem: 131693440 total, 131096312 free KiB Swap: 0 total, 0 free Timestamp of repository gentoo: Sun, 19 Jul 2020 17:30:01 +0000 Head commit of repository gentoo: 5afec32da5e0df9269375dd0cbf262fd0fe6ff32 sh bash 5.0_p17 ld GNU ld (Gentoo 2.33.1 p2) 2.33.1 distcc 3.3.3 x86_64-pc-linux-gnu [disabled] app-shells/bash: 5.0_p17::gentoo dev-java/java-config: 2.3.1::gentoo dev-lang/perl: 5.30.3::gentoo dev-lang/python: 2.7.18::gentoo, 3.6.11-r1::gentoo, 3.7.8-r1::gentoo, 3.8.3-r1::gentoo dev-util/cmake: 3.16.5::gentoo sys-apps/baselayout: 2.6-r1::gentoo sys-apps/openrc: 0.42.1::gentoo sys-apps/sandbox: 2.18::gentoo sys-devel/autoconf: 2.69-r4::gentoo sys-devel/automake: 1.16.1-r1::gentoo sys-devel/binutils: 2.33.1-r1::gentoo sys-devel/gcc: 9.3.0-r1::gentoo sys-devel/gcc-config: 2.3::gentoo sys-devel/libtool: 2.4.6-r6::gentoo sys-devel/make: 4.2.1-r4::gentoo sys-kernel/linux-headers: 5.4-r1::gentoo (virtual/os-headers) sys-libs/glibc: 2.30-r8::gentoo Repositories: gentoo location: /var/db/repos/gentoo sync-type: rsync sync-uri: rsync://rsync.gentoo.org/gentoo-portage priority: -1000 sync-rsync-verify-metamanifest: yes sync-rsync-verify-max-age: 24 sync-rsync-extra-opts: sync-rsync-verify-jobs: 1 local location: /var/db/repos/local masters: gentoo ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="@FREE" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=x86-64 -mtune=generic -O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-march=x86-64 -mtune=generic -O2 -pipe" DISTDIR="/var/cache/distfiles" ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR" FCFLAGS="-march=x86-64 -mtune=generic -O2 -pipe" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-march=x86-64 -mtune=generic -O2 -pipe" GENTOO_MIRRORS="http://distfiles.gentoo.org" LANG="en_US.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j25" PKGDIR="/var/cache/binpkgs" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" USE="acl amd64 berkdb bzip2 cli crypt dri fortran gdbm iconv ipv6 libtirpc multilib ncurses nls nptl openmp pam pcre readline seccomp split-usr ssl tcpd unicode xattr zlib" ABI_X86="64" ADA_TARGET="gnat_2018" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-2" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" RUBY_TARGETS="ruby25" USERLAND="GNU" VIDEO_CARDS="amdgpu fbdev intel nouveau radeon radeonsi vesa dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Created attachment 649964 [details] Patched Ebuild (sys-cluster/ceph-14.2.10)
Created attachment 649966 [details] Patched Ebuild (sys-cluster/ceph-15.2.4)
Created attachment 649968 [details, diff] Patch for Ebuild (sys-cluster/ceph-14.2.10)
Created attachment 649970 [details, diff] Patch for Ebuild (sys-cluster/ceph-15.2.4)
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=2eae111954e16b5efa850296d4ebe53a13692c71 commit 2eae111954e16b5efa850296d4ebe53a13692c71 Author: Patrick McLean <patrick.mclean@sony.com> AuthorDate: 2020-07-20 20:10:41 +0000 Commit: Patrick McLean <chutzpah@gentoo.org> CommitDate: 2020-07-20 20:10:41 +0000 sys-cluster/ceph: Use RelWithDebInfo cmake build type (bug 733316) Closes: https://bugs.gentoo.org/733316 Copyright: Sony Interactive Entertainment Inc. Package-Manager: Portage-2.3.103, Repoman-2.3.23 Signed-off-by: Patrick McLean <chutzpah@gentoo.org> sys-cluster/ceph/{ceph-14.2.10.ebuild => ceph-14.2.10-r1.ebuild} | 3 +++ sys-cluster/ceph/{ceph-15.2.4.ebuild => ceph-15.2.4-r1.ebuild} | 3 +++ sys-cluster/ceph/files/ceph-14.2.10-missing-includes.patch | 2 +- 3 files changed, 7 insertions(+), 1 deletion(-)
