Don't attempt to remove hidden aspects from the model

This triggers an assert in Qt, due to invalid arguments to row removal methods.

Done together with Alexander Semke.