impl-port: only add the port when buffers are negotiated

To avoid crashes when the node is scheduled but buffer have been
cleared.

See #904