server: Clean up socket destruction

The code here is wrong, leaky, and inconsistent. We don't free,
unlink or clean up things when we should in every error path.

Centralize the data destruction so it's easier to keep track of
and easier to bug fix.