colorimetry: use 4x4 matrices for colorimetry transforms

This is so that offsets can be represented in the matrices and not just scaled and rotated coordinate systems

Split out of !4821. Still gotta figure out why that one doesn't entirely work as expected