impl-link: don't clear ouput buffers in all cases

When we can't set the allocated buffers on the input port, don't
clear the buffers because they might be in use by the output port.

Fixes #1078