12
How would you design parallel grep for huge JSONL files? (lemmy.world)
submitted 7 months ago by to c/programming@programming.dev
24 comments fedilink hide all child comments
 

I need to scan very large JSONL files efficiently and am considering a parallel grep-style approach over line-delimited text.

Would love to hear how you would design it.

you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 7 months ago*

Could u use an already parallelized solution like ripgrep? I think someone else also mentioned putting it in a database, that shouldnt be too bad either.

  • permalink
  • fedilink
  • source
    •