backends: Prefer Output::modeSize() over Output::pixelSize()

Output transforms are handled differently, so let's use modeSize() to pick the buffer size consistently.