Do not cache already searched fprs for p15

This might lead to unexpected problems during testing
when users delete keys plug out / plug in smartcards etc.
An ldap query for a fingerprint is reasonable to do because
setCard is rarely called and should only be called when a new
card is detected.
2 jobs for master in 13 minutes and 58 seconds (queued for 5 seconds)
Status Name Job ID Coverage
  Build
passed build_clazy_clang_tidy #107008

00:09:39

passed linux kf5-qt5 qt5.15 #107007

00:04:19

0.0%