Worker thread count is unsigned

There should never be a negative amount of worker threads. Sematically this
makes sense and solves a lot of casting issues.