Me compressing binary data
comments
compress ze file
Yeah, I found zst recently as well. Quite a bit faster than xz/7z which is what I previously used (or gz when I just needed something fast).
And .tar.zst into 22!?
What was that log file? Millions of identical characters?
Am I reading that correctly that test.log.7z compressed 35 mB of text into 32 bytes?
It’s more relevant to the previous comment as an example of how we are doing a lot of compression at the filesystem level.
The files that are typically largest are already quite thoroughly compressed.
I don't see how that relates to encryption.
And then comes AV1, with crazily varying quality/compression for different source materials.
haha. I spent so much time looking for an efficient compression algorithm for audio and movies ... until I finally understood there are already compressed X) However it allow me to discover zst, wich is uncredible with text :
35M test.log
801K test.7z
32 test.log.7z
46 test.log.tar.bz2
45 test.log.tar.gz
108 test.log.tar.xz
22 test.log.tar.zst
2,1M test.tar.bz2
8,1M test.tar.gz
724K test.tar.xz
1,1M test.tar.zst
8,1M test.zip
We just need to develop an algorithm to compress random byte streams, easy
Media files can can benefit from a codec change. Going from h264 to h265/hevc can net a 30-50% reduction in size for almost no quality loss.
The only trade off is increased cpu usage if the client doesnt have hardware h265 support and the time to do the transcoding.
Ah, thanks. Figured it was just an edit of the climax of the movie. You know the one
Media files are always thoroughly compressed (except in certain settings like professional video and audio work).
It’s from the 2009 movie, during the montage where he is on Mars and is recounting to himself how he got his powers and his relationship with Janey. Timestamp is 1:10:08.
I believe it was a short clip shown when they went into Dr. Manhattans back story.
I actually just switched a backup script of mine from using a tar.gz to just a regular tar file. It's a little bigger but overall, the process is so much faster I don't even care about the tiny extra bit of compression (100gb vs 120gb transferred over a 1gbit connection). The entire reason I do this is, like you said, transferring files over the Internet is a billion times faster as one file, BUT you don't need the gzip step just for that
I dont remember him crushing a tank, is this an edit or did they release another watchmen movie?
Meh, that barely fits an empty MS Word doc...
/s
Same, and I feel like this still applies pretty well to arguments about AI bias not being something anyone should be too concerned about.
Well, yeah. The real advantage is only having a single file to transfer, makes eg. SFTP a lot less annoying at the command line.
Lossless compression works by storing redundant information more efficiently. If you've got 50 GB in a directory, it's going to be mostly pictures and videos, because that would be an incredible amount of text or source code. Those are already stored with lossy compression, so there's just not much more you can squeeze out.
I suppose you might have 50 GB of logs, especially if you've a logserver for your network? But most modern logging stores in a binary format, since it's quicker to search and manipulate, and doesn't use up such a crazy amount of disk space.
@irelephant@lemmy.dbzer0.com l thought this is a community for asking IT related questions. But which community would be more suitable for the question that l have asked ???
Turns out pseudo random byte streams don't really repeat that often.
Plot twist: Your file manager reports directories in GB and files in GB converted to GiB with GB as units
/j