Don't let transform tool use non-affine transforms on vector layers

The stroke checked the mode on its start, but it was possible to enter
invalid state by switching the mode after starting of the stroke.

Now the stroke is cancelled if the mode has changed after its start
into invalid value.

BUG:420919,419758