Do not cache already searched fprs for p15

This might lead to unexpected problems during testing
when users delete keys plug out / plug in smartcards etc.
An ldap query for a fingerprint is reasonable to do because
setCard is rarely called and should only be called when a new
card is detected.
2 jobs for master in 13 minutes and 58 seconds (queued for 5 seconds)
Status Name Job ID Coverage
passed build_clazy_clang_tidy #107008


passed linux kf5-qt5 qt5.15 #107007