Thank you for the quick reply. Adding pinentry-program /usr/bin/pinentry-qt5 to the ~/.gnupg/gpg-agent.conf and reloading the agent via echo RELOADAGENT | gpg-connect-agent solved the problem.
Example of the graphical error message.
Encrypting/signing message with non-smartcard key fails with empty error message