Fix double remove from surface destroy_listener_list

We remove the listener when a device loses its pointer focus, but
doesn't insert it in another destroy_listener list if surface is NULL.
When we set another pointer focus surface, we remove it again.
Tracked down and tested by Iskren Chernev.