Have a look at the 'Short Answer' in this substack on 'chunksize' in Python multiprocessing, then appreciate the sublime humour of the following comment to be found some 15-20 pages down...'This is one of the most epic answers on SO. You only need to look here to see that multiprocessing is super easy, barely an inconvenience.' stackoverflow.com/questions/53#it #technology #python #programming

you are viewing a single comment's thread
view the rest of the comments
[–] [S] 1 point 1 month ago

As an aside, chunksize is an easily-set integer arg to Python's pool function.Anything > 1 slows my app down, simply because the cycles performed in a typical large-job worker loop (into the billions) far exceed the frequency at which the loop returns a result, which is once, at the end.The problem I've got is while multi loops run fast, they tend to slow over time to about 50% of their startup speed. The end result is 4 cores vs. 1 cuts job time only ~50% vs. the expected 75% at job start.

  • source