Add a few tests to find out when exactly we're creating new config (or
when we should do that)