I need to scan very large JSONL files efficiently and am considering a parallel grep-style approach over line-delimited text.

Would love to hear how you would design it.

you are viewing a single comment's thread
view the rest of the comments
[+] 7 points 7 months ago* (last edited 7 months ago) (8 children)
  • [–] 3 points 7 months ago (7 children)

    Can a file really be split efficiently? And is reading from multiple files on the same disk really faster than scanning a single file from top to bottom?

  • source
  • parent
  • hideshow 7 child comments
  • [–] 6 points 7 months ago (1 child)

    You don't actually need to "split" anything, you just read from different offsets per thread. Mmap might be the most efficient way to do this (or at least the easiest)

    Whether or not that's going to run into hardware bottlenecks is a separate issue from designing a parallel algorithm. Idk what OP is trying to accomplish, but if their hardware is known (eg this is an internal tool meant to run in a data center), they'll need to read up on their hardware and virtualization architecture to squeeze the most IO performance.

    But if parsing is actually the bottleneck, there's a lot you can do to optimize it in software. Simdjson would be a good place to start.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 2 points 7 months ago (1 child)

    I think, you could open the same file multiple times and then just skip ahead by some number of bytes before you start reading.

    But yeah, no idea if this would actually be efficient. The bottleneck is likely still the hard drive and trying to fit multiple sections of the file into RAM might end up being worse than reading linearly...

  • source
  • parent
  • hideshow 1 child comment
  • load more comments (1 reply)