Fix terminal panel not keeping up with dir changes

The terminal panel is supposed to show the same location as the
currently active Dolphin view at all times.

However there was an issue when the terminal is supposed to
quickly switch to a new location and then back again to the old
one. The terminal ignored the switch to the old location unless it
had already fully switched to the new location. Because it isn't
particularly fast at fully switching to the new location, it would
never do the expected thing of switching back to the old location.

This commit makes it so the switch to the old location is only
ignored if there are no in-progress switches to a different
location.

BUG: 391380

BUG: 416690

FIXED-IN: 22.04.2

Not totally sure if this fixes everything but it seems like an improvement.