Only run one config file for now, we'll need to figure out how to make
the whole restarting logic work better, but it's not a common usecase,
so low priority for now.