dialog: Make sure that max and min sizes are always consistent

In Qt6 changing minimum and maximum sizes immediately changes the window size. This can lead to situations where upon resize the new minimum size is bigger than the current maximum size or the new maximum size smaller than the current minimum one. Going down this code path asserts. To avoid this, first set the maximum to the maximum of all sizes then the new minumum size and finally the new maximum.