backends: add support for custom modes in virtual outputs

It can be useful for autotests and for remote desktop use cases.

While this is technically speaking a feature and we're past the feature freeze, it's a really small and simple feature, so I think putting it into 6.6 is completely fine.