XWL only clears up when a "visit" finishes.

If we end a WlToXDrag through clicking we call "WlToXDrag::end" but the
visit object stays alive, which obviously will never finish.

Both the visit and the drag object then effectively leak and are used on
the next drag.

(cherry picked from commit 4bab721d)