xwayland: Do not remove X11Source while drag is going on
setWlSource will delete the X11Source while XToWlDrag did not finish leading to potential crashes. Since it's only set when creating a WlToXDrag, only remove it in this case. We can also replace the virtual with a type check if we have to do it anyways which also makes it clearer what is going on.