input: don't update the last usage serial for modifier keys

Xaver Huglrequested to merge
work/zamundaaa/modifier-serial into master

They're often used for shortcuts and don't necessarily indicate that the user is interacting with the window

cc #186

Merge request reports