Idle Worker Timeout doesn't work for single process workers / workers without semaphores

[dodamih]

I introduced a bug when adding PoolActivityTracker where the Tracker doesn't start unless you are using multiprocessed worker that has semaphores. I will fix this when I have time.