It looks like my journey to find a linux distribution where gpg and ssh authentication with a gpg smartcard works right out of the box is finally over.
Thanks to Moritz Bartl from torservers.net who pointed me in the direction to Debian testing aka stretch.
All I did was
sudo aptitude install pcscd pcsc-tools gpgsm gnupg-agent scdaemon -y
echo "use-agent" >> ~/.gnupg/gpg.conf
echo "enable-ssh-support" >> ~/.gnupg/gpg-agent.conf
Commenting out use-ssh-agent in /etc/X11/Xsession.options
sudo reboot
After that the card worked out of the box even with ssh auth on my servers. Without removing gnome-keyring(-daemon), removing whatsoever startscript, adding gpg-agent to .bashrc or whatever – it just works for now.
Hopefully it will continue to work 🙂