MAMEWorld >> EmuChat
View all threads Index   Flat Mode Flat  

lharms
MAME Fan
Reged: 01/07/06
Posts: 908
Send PM
Re: Can it beat t7z though?
10/10/16 08:29 AM


There are plenty of compression progs out there that beat 7zip (and t7z with its fixed parameter set). Usually by 5-20% in size. 7zip right now is the 'best balance' for the sort of thing going on in MAME. Speed and size wise. There are some out there that do much better on different types of data. For something like a streaming compression 7zip/deflate is actually a poor choice as it uses too much CPU and memory. There are much better ones out there for that choice. There are also tons of experimental ones out there. 7zip ended up being chosen because it seemed to be popular enough and was opensource enough to include.

Here is a decent recent comparison of the progs/algs http://mattmahoney.net/dc/text.html zstd is the one to look for. It seems to have ~15% worse compression ratio, similar compression times to 7zip, but wildly better decompress times. Its author seems to have designed it for viewtimes in programs. It compresses better than zlib so it is probably looking to be a drop in replacement for that. I would not be surprised at all if there are cases where it beats lzma. But in general size wise lzma seems to win.

Would it be worth adding? Maybe, if you want to optimize to speed of load. But it will cost you in size if you are currently using 7zip and lzma compressed stuff. But you would probably gain some space if you are using zip archives.

Also downside to the original site is he compares to zlib. Which is basically the baseline. It is usually not even as good as pkzip which it emulates. zlib is basically mid 90s 'ok' compression tech. Everyone compares to it because everyone seems to use it.

I personally am not unhappy with the decompression speed/size in MAME. What probably needs to be looked would be something in the rom lookup. From the procmon traces I see it looks to be doing a lot of back and forth opening files/directories and ignoring past results and repeating them. But that is just speculation on my part with a small bit of evidence. Have to dig into the code a bit and see.

Also CHDs is what consumes the most out of everything these days. That would probably be better served if CHDs could inherit from each other like VDI/VHD files do with x86 virtual machines. Sort of like what happened with merged sets a long time ago. Which in itself yielded about a 50% savings. Even then it may not be interesting in the amount saved or work at all in saving anything due to the way these sorts of devices lay data out.

The crux of the issue are there better compressors out there worth using? Lets say you get a modest 5% better compression. That would be well over 100 gig of data. Nothing to sneeze at. But not as interesting when you consider the whole thing stands around 2.5TB right now.







Entire thread
Subject Posted by Posted on
* worth a look?: smaller-and-faster-data-compression-with -zstandard RdW 10/04/16 08:22 PM
. * Can it beat t7z though? Dullaron  10/09/16 11:03 AM
. * Re: Can it beat t7z though? lharms  10/10/16 08:29 AM
. * Re: Can it beat t7z though? R. Belmont  10/11/16 07:25 PM
. * Re: Can it beat t7z though? lharms  10/12/16 01:36 AM
. * Re: worth a look?: smaller-and-faster-data-compression-with -zstandard Roman  10/04/16 09:26 PM
. * Re: worth a look?: smaller-and-faster-data-compression-with -zstandard RdW  10/05/16 10:33 PM
. * Re: worth a look?: smaller-and-faster-data-compression-with -zstandard R. Belmont  10/06/16 04:54 PM
. * Re: worth a look?: smaller-and-faster-data-compression-with -zstandard TafoidAdministrator  10/05/16 11:03 PM
. * Re: worth a look?: smaller-and-faster-data-compression-with -zstandard Haze  10/05/16 10:47 PM
. * Re: worth a look?: smaller-and-faster-data-compression-with -zstandard Master O  10/05/16 01:57 AM
. * Re: worth a look?: smaller-and-faster-data-compression-with -zstandard SmitdoggAdministrator  10/05/16 02:15 AM
. * Re: worth a look?: smaller-and-faster-data-compression-with -zstandard smf  10/06/16 01:50 PM
. * Re: worth a look?: smaller-and-faster-data-compression-with -zstandard Haze  10/05/16 03:08 AM
. * Re: worth a look?: smaller-and-faster-data-compression-with -zstandard Master O  10/05/16 03:22 AM
. * Re: worth a look?: smaller-and-faster-data-compression-with -zstandard Haze  10/05/16 03:28 AM
. * Re: worth a look?: smaller-and-faster-data-compression-with -zstandard MooglyGuy  10/05/16 09:34 AM
. * Re: worth a look?: smaller-and-faster-data-compression-with -zstandard Haze  10/05/16 09:31 PM
. * Re: worth a look?: smaller-and-faster-data-compression-with -zstandard R. Belmont  10/06/16 05:04 PM
. * Re: worth a look?: smaller-and-faster-data-compression-with -zstandard Traso  10/05/16 06:44 PM
. * Re: worth a look?: smaller-and-faster-data-compression-with -zstandard MooglyGuy  10/05/16 07:55 PM
. * Re: worth a look?: smaller-and-faster-data-compression-with -zstandard Traso  10/09/16 02:51 AM

Extra information Permissions
Moderator:  Robbbert, Tafoid 
0 registered and 185 anonymous users are browsing this forum.
You cannot start new topics
You cannot reply to topics
HTML is enabled
UBBCode is enabled
Thread views: 2544