I use libressl since rb_libtorrent-1.0.7 for a backend with qbittorrent using OPTION "require encryption" not recognizing any problem.
@Ulenrich Are you using a custom ebuild too? Can you provide it? :)
Created attachment 432330 [details] Custom ebuild for supporting LibreSSL. Patched ebuild here based on upstream rb_libtorrent-1.0.9-r1.ebuild
Fixed in 1.0.9-r2 https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=9a7268d86d008c69d0f0ccbb3cda4ed455febdba