The internal MIDI player (enabled by the `portmidi' USE flag) of media-sound/frescobaldi requires the `pyportmidi' python module, which can be introduced by the `python' USE flag of media-libs/portmidi. Therefore, media-sound/frescobaldi, when enabling the portmidi USE flag, should depend on `media-libs/portmidi[python]', not just `media-libs/portmidi' (as of media-sound/frescobaldi-2.0.4).
I believe you are mistaken as per upstream INSTALL file: For MIDI the PortMidi library is used, either via importing 'pypm', 'pyportmidi._pyportmidi, or [ ... ] embedding the PortMidi C-library via ctypes. And the C-library is always installed, so the fallback is available in any case.
Yes, now I find it working without media-libs/portmidi[python], so something might be wrong with my configurations at that time. Thanks for your clarification :)