colorimetry: use 4x4 matrices for colorimetry transforms
This is so that offsets can be represented in the matrices and not just scaled and rotated coordinate systems
Split out of !4821 (closed). Still gotta figure out why that one doesn't entirely work as expected