When upgrading to Ceph Quincy I was running into errors accessing RBD until I set the osd_class_dir manually in ceph.conf. Reproducible: Always Steps to Reproduce: 1. upgrade to ceph 17.2.0-r1 2. try to access RBD (rbd du --pool some_pool some_image) Actual Results: 2022-05-05T19:09:28.202+0000 7f6c86f41280 -1 librbd::api::Trash: list_trash_image_specs: error listing rbd trash entries: (95) Operation not supported 2022-05-05T19:09:28.202+0000 7f6c7f7fe640 -1 librbd::image::OpenRequest: failed to retrieve image id: (95) Operation not supported 2022-05-05T19:09:28.202+0000 7f6c7ffff640 -1 librbd::ImageState: 0x55775e381960 failed to open image: (95) Operation not supported rbd: error opening steam: (95) Operation not supported NAME PROVISIONED USED rbd: du failed: (95) Operation not supported Expected Results: NAME PROVISIONED USED some_image 8 TiB 2.0 TiB After upgrading to quincy I got errors like these whenever I tried to access RBD: ==> /var/log/ceph/ceph-osd.2.log <== 2022-05-05T19:11:03.042+0000 7fa845ab7640 0 _load_class could not stat class lib64/rados-classes/libcls_rbd.so: (2) No such file or directory As can be seen the path it tries to access is `lib64/rados-classes/libcls_rbd.so`. The correct path would've been `/usr/lib64/rados-classes/libcls_rbd.so`. As per upstream documentation (https://docs.ceph.com/en/pacific/rados/configuration/osd-config-ref/) the default for osd_class_dir is $libdir/rados-classes. My current ceph.conf looks like this and everything works again: [global] fsid = […] mon_host = […] osd_class_dir = /usr/lib64/rados-classes I'm assuming this is a Gentoo specific issue as I was literally unable to find any information on this on search engines and the Ceph update was only recently added to the portage tree, however it might also be an upstream issue, libdir settings just seemed like a Gentoo thing.
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=5ae443701f3ac7d74af946097041380e9a8e2014 commit 5ae443701f3ac7d74af946097041380e9a8e2014 Author: Patrick McLean <chutzpah@gentoo.org> AuthorDate: 2022-05-05 22:48:31 +0000 Commit: Patrick McLean <chutzpah@gentoo.org> CommitDate: 2022-05-05 22:51:36 +0000 sys-cluster/ceph: Fix rados-classes path (bug #842822) With newer versions of cmake, CMAKE_INSTALL_LIBDIR is no longer an absolute path. This made the search path for the rados-classes be a relative path, which did not work. Switch to using CMAKE_INSTALL_FULL_LIBDIR instead. Closes: https://bugs.gentoo.org/842822 Signed-off-by: Patrick McLean <chutzpah@gentoo.org> .../ceph/{ceph-17.2.0-r1.ebuild => ceph-17.2.0-r2.ebuild} | 1 + sys-cluster/ceph/files/ceph-17.2.0-osd_class_dir.patch | 13 +++++++++++++ 2 files changed, 14 insertions(+)
