Also do some cleanup. As far as I can see the custom rpath and CFLAG magic should not be need as Craft will deal with that
(cherry-picked from 57b4cd20)
cc @jlskuz