CLI-based plugins run their jobs in another process, so we can avoid to also
run them in another thread. This also fixes a race condition with the unit
tests.

Differential Revision: D1229