Have a look at the 'Short Answer' in this substack on 'chunksize' in Python multiprocessing, then appreciate the sublime humour of the following comment to be found some 15-20 pages down...'This is one of the most epic answers on SO. You only need to look here to see that multiprocessing is super easy, barely an inconvenience.' stackoverflow.com/questions/53#it #technology #python #programming

you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 1 month ago

@f800gecko Chunksize changes how work gets handed to workers, not how long each task takes. I've seen a bad value leave processes idle while the last batch finishes.

  • source