screencast: Fix format announced with SHM enum format

Kwin announces a format whith alpha when Dma-Bufs are available, even when the texture doesn't contain any. This results in clients segfault when trying to acces the buffer assuming a maxsize wrt. the announced format by ways of dimensions and bpp.

This patch moves the format masking to affect the Dma-Buf transport only.

CC: @apol