I need to scan very large JSONL files efficiently and am considering a parallel grep-style approach over line-delimited text.

Would love to hear how you would design it.

you are viewing a single comment's thread
view the rest of the comments
[–] 2 points 7 months ago (1 child)

I think, you could open the same file multiple times and then just skip ahead by some number of bytes before you start reading.

But yeah, no idea if this would actually be efficient. The bottleneck is likely still the hard drive and trying to fit multiple sections of the file into RAM might end up being worse than reading linearly...

  • source
  • parent
  • hideshow 1 child comment
  • load more comments (1 reply)