Revert "libdrm: Fix issue about differrent domainID but same BDF"

This reverts commit 56c21f87.

There were issues pointed out during review that were not addressed.
Would love to have this re-land, once those are addressed.