audioadapter: emit Latency param changed when follower changed

When the follower port Emits a latency param changed, our reported
latency also changes so we need to emit the changed notify.