If we're installed in multiple prefixes (ie self built vs system packages)
only load the proper one which is from the prefix of the self built one