XWL only clears up when a "visit" finishes.

If we end a WlToXDrag through clicking we call "WlToXDrag::end" but the
visit object stays alive, which obviously will never finish.

Both the visit and the drag object then effectively leak and are used on
the next drag.