Created attachment 928424 [details] energe --info sys-cluster/ceph I've upgraded my ceph test system from 19.2.2 to 19.2.2-r1 and discovered that one of mgr python is broken again. 2025-05-10T22:57:52.315+0200 7effc8c63100 1 mgr[py] Loading python module 'restful' 2025-05-10T22:57:52.438+0200 7effc8c63100 -1 mgr[py] Module not found: 'restful' 2025-05-10T22:57:52.438+0200 7effc8c63100 -1 mgr[py] Traceback (most recent call last): File "/usr/share/ceph/mgr/restful/__init__.py", line 1, in <module> from .module import Module File "/usr/share/ceph/mgr/restful/module.py", line 22, in <module> from OpenSSL import crypto File "/lib/python3.13/site-packages/OpenSSL/__init__.py", line 8, in <module> from OpenSSL import SSL, crypto File "/lib/python3.13/site-packages/OpenSSL/SSL.py", line 15, in <module> from cryptography import x509 File "/lib/python3.13/site-packages/cryptography/x509/__init__.py", line 7, in <module> from cryptography.x509 import certificate_transparency, verification File "/lib/python3.13/site-packages/cryptography/x509/certificate_transparency.py", line 8, in <module> from cryptography.hazmat.bindings._rust import x509 as rust_x509 ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process Error ENOENT: module 'restful' reports that it cannot run on the active manager daemon: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process (pass --force to force enablement)
Dupe of bug 920906? I don't think the 19.2.2 vs 19.2.2-r1 is it, I think it's more likely something else changed? *** This bug has been marked as a duplicate of bug 920906 ***
If downgrading only ceph really works, then... $ diff -u ceph-19.2.2.ebuild ceph-19.2.2-r1.ebuild --- ceph-19.2.2.ebuild 2025-05-09 11:12:00.907644326 +0100 +++ ceph-19.2.2-r1.ebuild 2025-05-09 11:12:00.907644326 +0100 @@ -3,7 +3,7 @@ EAPI=8 -PYTHON_COMPAT=( python3_{10..12} ) +PYTHON_COMPAT=( python3_{10..13} ) LUA_COMPAT=( lua5-{3..4} ) inherit check-reqs bash-completion-r1 cmake flag-o-matic lua-single \ @@ -241,6 +241,10 @@ "${FILESDIR}/ceph-19.2.1-exporter.patch" "${FILESDIR}/ceph-19.2.1-isa-l.patch" "${FILESDIR}/ceph-19.2.1-quiet-stringop-truncation.patch" + "${FILESDIR}/ceph-19.2.2-py313-1.patch" + "${FILESDIR}/ceph-19.2.2-py313-2.patch" + "${FILESDIR}/ceph-19.2.2-py313-3.patch" + "${FILESDIR}/ceph-19.2.2-gcc15.patch" ) check-reqs_export_vars() { But please verify that.
I don't know if that's a duplicate of 920906 bug 920906 hasn't been an issue for me for a long time on 19.2.1 and 19.2.2. I assumed it was resolved. I can enabled mgr module dashboard and restful on my 19.2.2 cluster without issues. Both ceph clusters (main and test) have the same bcrypt and cryptography versions. dev-python/cryptography-44.0.2 dev-python/bcrypt-4.3.0 I'll try to recompile and test 19.2.2-r1 without these 4 patches since I can't just reinstall old binpkg because of python_compat
I've tried to recompile sys-cluster/ceph-19.2.2-r1 without new patches but it doesn't compile. I've compiled sys-cluster/ceph-19.2.2-r1 with all 3 patches but for python 3.12 (USE=python_targets_python3_12) And on python 3.12 mgr works just fine and module restful just works. Example from the log 2025-05-11T14:07:53.472+0200 7f93dd4de100 1 mgr[py] Loading python module 'selftest' 2025-05-11T14:07:53.526+0200 7f93dd4de100 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2025-05-11T14:07:53.526+0200 7f93dd4de100 1 mgr[py] Loading python module 'rgw' 2025-05-11T14:07:53.646+0200 7f93dd4de100 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2025-05-11T14:07:53.646+0200 7f93dd4de100 1 mgr[py] Loading python module 'restful' 2025-05-11T14:07:53.792+0200 7f93dd4de100 1 mgr[py] Loading python module 'rbd_support' 2025-05-11T14:07:53.866+0200 7f93dd4de100 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2025-05-11T14:07:53.866+0200 7f93dd4de100 1 mgr[py] Loading python module 'prometheus' 2025-05-11T14:07:54.042+0200 7f93dd4de100 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2025-05-11T14:07:54.042+0200 7f93dd4de100 1 mgr[py] Loading python module 'progress' Could you reconsider un-resolving this bug? It seems to be specific to this python 3.13 upgrade.
Mysterious. Thanks for poking.
I'm sorry, I've made a mistake. I've actually compiled and tested 19.2.2 not 19.2.2-r1 on python 3.12 Now I've actually compiled 19.2.2-r1 on python 3.12 still has this issue. The issue is probably caused my 4 new patches in 19.2.2-r1 error ceph mgr module enable restful Error ENOENT: module 'restful' reports that it cannot run on the active manager daemon: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process (pass --force to force enablement)
(In reply to ev from comment #6) > I'm sorry, I've made a mistake. > > I've actually compiled and tested 19.2.2 not 19.2.2-r1 on python 3.12 > > Now I've actually compiled 19.2.2-r1 on python 3.12 still has this issue. > The issue is probably caused my 4 new patches in 19.2.2-r1 > > error > ceph mgr module enable restful > Error ENOENT: module 'restful' reports that it cannot run on the active > manager daemon: PyO3 modules compiled for CPython 3.8 or older may only be > initialized once per interpreter process (pass --force to force enablement) Can you look at which patch causes it?
I've tried to look into it again and tried some proxmox patches but no luck. I'm not sure which patch causes it. These 3 patches may as well be considered a single patch. You cannot use them separately. ceph-19.2.2-py313-1.patch ceph-19.2.2-py313-2.patch ceph-19.2.2-py313-3.patch In general this issue affects modules restful and dashboard and completely break ceph-mgr ability to export prometheus metrics. 2025-05-31T14:37:00.304+0200 7f623d099100 1 mgr[py] Loading python module 'restful' 2025-05-31T14:37:00.561+0200 7f623d099100 -1 mgr[py] Module not found: 'restful' 2025-05-31T14:37:00.561+0200 7f623d099100 -1 mgr[py] Traceback (most recent call last): File "/usr/share/ceph/mgr/restful/__init__.py", line 1, in <module> from .module import Module File "/usr/share/ceph/mgr/restful/module.py", line 22, in <module> from OpenSSL import crypto File "/lib/python3.13/site-packages/OpenSSL/__init__.py", line 8, in <module> from OpenSSL import SSL, crypto File "/lib/python3.13/site-packages/OpenSSL/SSL.py", line 15, in <module> from cryptography import x509 File "/lib/python3.13/site-packages/cryptography/x509/__init__.py", line 7, in <module> from cryptography.x509 import certificate_transparency, verification File "/lib/python3.13/site-packages/cryptography/x509/certificate_transparency.py", line 8, in <module> from cryptography.hazmat.bindings._rust import x509 as rust_x509 ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process 2025-05-31T14:34:52.434+0200 7f0485883100 1 mgr[py] Loading python module 'cephadm' 2025-05-31T14:34:52.934+0200 7f0485883100 -1 mgr[py] Module not found: 'cephadm' 2025-05-31T14:34:52.934+0200 7f0485883100 -1 mgr[py] Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/__init__.py", line 1, in <module> from .module import CephadmOrchestrator File "/usr/share/ceph/mgr/cephadm/module.py", line 40, in <module> from cephadm.http_server import CephadmHttpServer File "/usr/share/ceph/mgr/cephadm/http_server.py", line 6, in <module> from cephadm.agent import AgentEndpoint File "/usr/share/ceph/mgr/cephadm/agent.py", line 23, in <module> from cephadm.ssl_cert_utils import SSLCerts File "/usr/share/ceph/mgr/cephadm/ssl_cert_utils.py", line 8, in <module> from cryptography import x509 File "/lib/python3.13/site-packages/cryptography/x509/__init__.py", line 7, in <module> from cryptography.x509 import certificate_transparency, verification File "/lib/python3.13/site-packages/cryptography/x509/certificate_transparency.py", line 8, in <module> from cryptography.hazmat.bindings._rust import x509 as rust_x509 ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process Can someone with more python experience look into this?
Given the mess in bug 920906 (where, fundamentally, it's all workarounds until it's properly fixed upstream), I don't think we can do much.
