you are viewing a single comment's thread
view the rest of the comments
[–] 5 points 1 year ago (2 children)

I’d probably just use line delimited JSON or CSV for this use case. It plays nicely with cat and other standard tools and basically all the yaml is doing is wrapping raw json and adding extra parse time/complexity.

In the end consider converting this to parquet for analysis, you probably won’t get much from compression or row-group clustering, but you will get benefits from the column store format when reading the data.

  • source
  • hideshow 2 child comments