12
How would you design parallel grep for huge JSONL files? (lemmy.world)
submitted 7 months ago by to c/programming@programming.dev
24 comments fedilink hide all child comments
 

I need to scan very large JSONL files efficiently and am considering a parallel grep-style approach over line-delimited text.

Would love to hear how you would design it.

you are viewing a single comment's thread
view the rest of the comments
[–] 4 points 7 months ago (1 child)
  1. How many grep-like ops per file?
  2. Is it interactive or run by another process?
  3. Do you know which files ahead of time?
  4. Do you have any control over that file creation?
  5. Is the JSONL append only? Is the grep running while the file is modified?
  6. How large is very large? 100s of MB? Few GB? 100s of GB? Whether or not it fits in memory could change the approach.
  7. You're using files, plural, would parallelizing at the file level (e.g. one thread per file) be enough?
  8. How many files and how often is that executed?
  • permalink
  • fedilink
  • source
  • hideshow 1 child comment
    •