This doesn't solve all the bug with the back button but at least the one when controlsVisible is true.
The remaining bug could probably be fixed in another MR with a shadow: