12
How would you design parallel grep for huge JSONL files? (lemmy.world)
submitted 7 months ago by to c/programming@programming.dev
24 comments fedilink hide all child comments
 

I need to scan very large JSONL files efficiently and am considering a parallel grep-style approach over line-delimited text.

Would love to hear how you would design it.

you are viewing a single comment's thread
view the rest of the comments
[–] 2 points 7 months ago

I think mmap is unlikely to be the best option seeing as you'd be doing large sequential reads.

  • permalink
  • fedilink
  • source
  • parent