audioconvert: don't just copy the converter flags

We need to manage our own flags based on the direction of the follower.

We also need to make sure we only clear the NEED_CONFIGURE flag
after we are actually configured, not just when we configured our
internal converter.

See #1548