Default TBB threadpool may be too aggressive for downloads #155
Description
Currently the default TBB threadpool can be too aggressive for curl downloads.
When developing both the Aeryn OS and Solus backends for the CPP rewrite I often hit timeouts when testing against remote suites. These timeouts happened mostly against larger packages (100MB+). Despite the retry mechanism, the large downloads often failed to complete stopping the generation run.
As a last resort, I nuked the TBB pool and converted it to a sequential for-loop to finish testing remote suite capabilities.
#152 helps a lot by resuming the download instead of re-downloading from scratch but it isn't necessarily the best overall solution.
In the case of a machine with many threads against a low-bandwidth remote we may want to limit the number of concurrent downloads somehow.
Possibly, a good strategy is to create a separate TBB pool for remote downloads that'll start off at 3 jobs and can increase/decrease concurrency if throughput remains good. But of course, it'll add a decent amount of complexity.