audioadapter: fix passthrough switching some more

Remove the follower ports when leaving passthrough, add the DSP
ports.
Set up the link_io between follower and convert again when leaving
passthrough.