Only add Qt module includes when needed, avoid unneeded compile flags

This makes debugging harder and causes verbose logs. Also, explicitly configuring some edgecases may avoid unintended special behavior