|
worth a look?: smaller-and-faster-data-compression-with -zstandard
#359335 - 10/04/16 08:22 PM
|
|
|
|
|
|
Re: worth a look?: smaller-and-faster-data-compression-with -zstandard
[Re: RdW]
#359341 - 10/04/16 09:26 PM
|
|
|
In days where multi terabyte hds are commonly used (and for MAME 1TB is more than enough for your pirated roms), compression ratio/speed does not really matter.
|
|
|
|
Re: worth a look?: smaller-and-faster-data-compression-with -zstandard
[Re: Roman]
#359357 - 10/05/16 01:57 AM
|
|
|
> In days where multi terabyte hds are commonly used (and for MAME 1TB is more than
> enough for your pirated roms), compression ratio/speed does not really matter.
"Mame wastes your disk space with impunity." - One of the Mamedevs; I don't remember who.
|
"Note to Noobs:
We are glad to help you but simply posting that something does not work is not going to lead to you getting help. The more information you can supply defining your problem, the less likely it will be that you will get smart-alec replies.
C.D.~"
|
|
|
Re: worth a look?: smaller-and-faster-data-compression-with -zstandard
[Re: Master O]
#359359 - 10/05/16 02:15 AM
|
|
|
Space maybe but speed definitely matters for future chd compression/use. There's a reason the original PC based arcade games didn't have their data all zipped.
|
|
|
|
Re: worth a look?: smaller-and-faster-data-compression-with -zstandard
[Re: Smitdogg]
#359363 - 10/05/16 03:08 AM
|
|
|
> Space maybe but speed definitely matters for future chd compression/use. There's a
> reason the original PC based arcade games didn't have their data all zipped.
that's not strictly true, plenty of games did have compressed data
there are also logical reasons to do it, if you're dealing with streamed data then you don't want i/o to become a bottleneck, so if you can stream compressed audio and decompress it you're using less i/o time than streaming uncompressd audio. For certain types of data it's also faster in general to load + decompress than load a larger block of data (eg large amounts of text, if you load a dictionary first, then only have to load lookup to words, it's a faster than reading raw data)
also drive access speeds improve all the time, the majority of arcade systems that are viable targets had slower drives anyway.
yeah, it could be an issue, but it's definitely not a black/white thing.
there are of course other interesting cases when it comes to i/o bandwidth, with GTA5 it's recommended you DON'T do a full HDD install on an Xbox 360, the logic behind this seems to be that you end up i/o bound streaming everything from the HDD, but if you stream some data from the DVD and other data from the HDD at the same time you have more i/o bandwidth.
|
|
|
|
Re: worth a look?: smaller-and-faster-data-compression-with -zstandard
[Re: Haze]
#359364 - 10/05/16 03:22 AM
|
|
|
> > Space maybe but speed definitely matters for future chd compression/use. There's a
> > reason the original PC based arcade games didn't have their data all zipped.
>
> that's not strictly true, plenty of games did have compressed data
>
> there are also logical reasons to do it, if you're dealing with streamed data then
> you don't want i/o to become a bottleneck, so if you can stream compressed audio and
> decompress it you're using less i/o time than streaming uncompressd audio. For
> certain types of data it's also faster in general to load + decompress than load a
> larger block of data (eg large amounts of text, if you load a dictionary first, then
> only have to load lookup to words, it's a faster than reading raw data)
>
> also drive access speeds improve all the time, the majority of arcade systems that
> are viable targets had slower drives anyway.
>
> yeah, it could be an issue, but it's definitely not a black/white thing.
>
> there are of course other interesting cases when it comes to i/o bandwidth, with GTA5
> it's recommended you DON'T do a full HDD install on an Xbox 360, the logic behind
> this seems to be that you end up i/o bound streaming everything from the HDD, but if
> you stream some data from the DVD and other data from the HDD at the same time you
> have more i/o bandwidth.
By HDD, do you mean the mechanical ones only?
How does that change with SSDs, then, if at all?
|
"Note to Noobs:
We are glad to help you but simply posting that something does not work is not going to lead to you getting help. The more information you can supply defining your problem, the less likely it will be that you will get smart-alec replies.
C.D.~"
|
|
|
Re: worth a look?: smaller-and-faster-data-compression-with -zstandard
[Re: Master O]
#359367 - 10/05/16 03:28 AM
|
|
|
> > > Space maybe but speed definitely matters for future chd compression/use. There's
> a
> > > reason the original PC based arcade games didn't have their data all zipped.
> >
> > that's not strictly true, plenty of games did have compressed data
> >
> > there are also logical reasons to do it, if you're dealing with streamed data then
> > you don't want i/o to become a bottleneck, so if you can stream compressed audio
> and
> > decompress it you're using less i/o time than streaming uncompressd audio. For
> > certain types of data it's also faster in general to load + decompress than load a
> > larger block of data (eg large amounts of text, if you load a dictionary first,
> then
> > only have to load lookup to words, it's a faster than reading raw data)
> >
> > also drive access speeds improve all the time, the majority of arcade systems that
> > are viable targets had slower drives anyway.
> >
> > yeah, it could be an issue, but it's definitely not a black/white thing.
> >
> > there are of course other interesting cases when it comes to i/o bandwidth, with
> GTA5
> > it's recommended you DON'T do a full HDD install on an Xbox 360, the logic behind
> > this seems to be that you end up i/o bound streaming everything from the HDD, but
> if
> > you stream some data from the DVD and other data from the HDD at the same time you
> > have more i/o bandwidth.
>
> By HDD, do you mean the mechanical ones only?
>
> How does that change with SSDs, then, if at all?
you can still end up limited by the speed of the drive controller etc.
for example putting an SSD in a PS3 seems to make absolutely no difference to performance even with games that are fully installed.
there are bottlenecks all over the place in systems.
I was reading a while back that one issue some devs have encountered in emulating video cards is that the ram on the video cards they're emulating is faster than the ram of the machines they're writing the emulator on, so even something as simple as a vram->vram copy done by the game will always be slower than 100% speed in the emulator because a simple memcpy isn't fast enough. Of course that only applies to the emulation of very recent systems.
|
|
|
|
Re: worth a look?: smaller-and-faster-data-compression-with -zstandard
[Re: Haze]
#359374 - 10/05/16 09:34 AM
|
|
|
> Of course that only applies to
> the emulation of very recent systems.
It applies even to systems that we don't think of as "very recent". I was reading the Nocash documentation on the PS1 just the other day, and I seem to recall that clear operations on the PS1's GPU will clear VRAM in blocks of 32 bytes per cycle. That's something that only really became viable on modern CPUs with the introduction of AVX extensions.
|
|
|
|
Re: worth a look?: smaller-and-faster-data-compression-with -zstandard
[Re: MooglyGuy]
#359385 - 10/05/16 06:44 PM
|
|
|
> It applies even to systems that we don't think of as "very recent". I was reading the Nocash documentation on the PS1...
Over twenty years old isn't remotely recent in game and hardware development.
|
Scifi frauds. SF illuminates.
_________________
Culture General Contact Unit (Eccentric)
|
|
|
Re: worth a look?: smaller-and-faster-data-compression-with -zstandard
[Re: Traso]
#359394 - 10/05/16 07:55 PM
|
|
|
> > It applies even to systems that we don't think of as "very recent". I was reading
> the Nocash documentation on the PS1...
>
> Over twenty years old isn't remotely recent in game and hardware development.
Yeah, maybe that's why I pointed out that it's not hardware that we think of as "very recent" anymore. Is reading not your strong suit, or were you just that desperate for a reason to be a sarcastic shithead?
|
|
|
|
Re: worth a look?: smaller-and-faster-data-compression-with -zstandard
[Re: MooglyGuy]
#359404 - 10/05/16 09:31 PM
|
|
|
> > Of course that only applies to
> > the emulation of very recent systems.
>
> It applies even to systems that we don't think of as "very recent". I was reading the
> Nocash documentation on the PS1 just the other day, and I seem to recall that clear
> operations on the PS1's GPU will clear VRAM in blocks of 32 bytes per cycle. That's
> something that only really became viable on modern CPUs with the introduction of AVX
> extensions.
a single cycle on the PSX I'm guessing is a longer period of time as the actual clocks are lower?
|
|
|
|
Re: worth a look?: smaller-and-faster-data-compression-with -zstandard
[Re: Roman]
#359410 - 10/05/16 10:33 PM
|
|
|
> In days where multi terabyte hds are commonly used (and for MAME 1TB is more than
> enough for your pirated roms), compression ratio/speed does not really matter.
Why was the support for 7z added in .146 then again?
|
|
|
|
Re: worth a look?: smaller-and-faster-data-compression-with -zstandard
[Re: RdW]
#359411 - 10/05/16 10:47 PM
|
|
|
> > In days where multi terabyte hds are commonly used (and for MAME 1TB is more than
> > enough for your pirated roms), compression ratio/speed does not really matter.
>
> Why was the support for 7z added in .146 then again?
because it could be
because it's a more widely used 64-bit capable container than zip64 (or at least was at the time)
because people were supplying dumps as .7z files, and it was useful to be able to work with them directly.
i'm also informed that some of the more recent pirate multigames have things like hardware decompression modules (likely custom programmed FPGAs) that decompress such things, so if we ever do support those we'll need the code there to be able to emulate the hardware. (allowing them to run .7z / .zip games from flash cards etc.)
|
|
|
Tafoid |
|
I keep on testing.. testing.. testing... into the future!
|
|
|
|
|
Reged: 04/19/06
|
|
Posts: 3135
|
|
Loc: USA
|
|
|
|
Send PM
|
|
|
Re: worth a look?: smaller-and-faster-data-compression-with -zstandard
[Re: RdW]
#359414 - 10/05/16 11:03 PM
|
|
|
> > In days where multi terabyte hds are commonly used (and for MAME 1TB is more than
> > enough for your pirated roms), compression ratio/speed does not really matter.
>
> Why was the support for 7z added in .146 then again?
I'm not entirely sure if your question was very serious or just to play the troll, but adding new compression algorithms isn't a move made lightly. At that time, 7z had finally served its time, was well tested, and had become pretty much the standard across all platforms. New compression algorithms, while they may promise a lot, need time to mature and become fully tested and also had become more mainstream favored. So, while it was added with CHD items in mind, regular romsets and softlist items can benefit now from this addition and be stored more efficiently.
|
|
|
| smf |
|
I've been here before
|
|
|
|
|
|
Reged: 01/16/15
|
|
Posts: 130
|
|
|
|
|
|
Send PM
|
|
|
Re: worth a look?: smaller-and-faster-data-compression-with -zstandard
[Re: Smitdogg]
#359441 - 10/06/16 01:50 PM
|
|
|
> Space maybe but speed definitely matters for future chd compression/use.
Some people already convert the chd's to "none" compressed files, because spikes in decompression speed throw the timing off on some rhythm games.
|
|
|
| R. Belmont |
|
Cuckoo for IGAvania
|
|
|
|
|
|
Reged: 09/21/03
|
|
Posts: 9713
|
|
Loc: ECV-197 The Orville
|
|
|
|
Send PM
|
|
|
Re: worth a look?: smaller-and-faster-data-compression-with -zstandard
[Re: RdW]
#359448 - 10/06/16 04:54 PM
|
|
|
> > In days where multi terabyte hds are commonly used (and for MAME 1TB is more than
> > enough for your pirated roms), compression ratio/speed does not really matter.
>
> Why was the support for 7z added in .146 then again?
Because we need to be able to support ROMs with more than 4 GB of data. 7zip exists on all the platforms we care about and has been well-tested. zip64 fails those criteria.
|
|
|
| R. Belmont |
|
Cuckoo for IGAvania
|
|
|
|
|
|
Reged: 09/21/03
|
|
Posts: 9713
|
|
Loc: ECV-197 The Orville
|
|
|
|
Send PM
|
|
|
Re: worth a look?: smaller-and-faster-data-compression-with -zstandard
[Re: Haze]
#359449 - 10/06/16 05:04 PM
|
|
|
> > > Of course that only applies to
> > > the emulation of very recent systems.
> >
> > It applies even to systems that we don't think of as "very recent". I was reading
> the
> > Nocash documentation on the PS1 just the other day, and I seem to recall that clear
> > operations on the PS1's GPU will clear VRAM in blocks of 32 bytes per cycle. That's
> > something that only really became viable on modern CPUs with the introduction of
> AVX
> > extensions.
>
> a single cycle on the PSX I'm guessing is a longer period of time as the actual
> clocks are lower?
PSX GPU clock is 53.693175 MHz. So 32 bytes per cycle works out to 1638 megabytes per second, or 27 MB / 60 Hz frame. That's definitely something even on modern hardware.
|
|
|
|
Re: worth a look?: smaller-and-faster-data-compression-with -zstandard
[Re: MooglyGuy]
#359542 - 10/09/16 02:51 AM
|
|
|
> > Over twenty years old isn't remotely recent in game and hardware development.
> Yeah, maybe that's why I pointed out that it's not hardware that we think of as "very recent" anymore. Is reading not your strong suit, or were you just that desperate for a reason to be a sarcastic shithead?
I think I read it....weird. I dunno.
|
Scifi frauds. SF illuminates.
_________________
Culture General Contact Unit (Eccentric)
|
|
| Dullaron |
|
Diablo III - Dunard #1884
|
|
|
|
|
Reged: 07/22/05
|
|
Posts: 6123
|
|
Loc: Fort Worth, Tx
|
|
|
|
Send PM
|
|
|
Can it beat t7z though?
[Re: RdW]
#359560 - 10/09/16 11:03 AM
|
|
|
|
W11 Home 64-bit + Nobara OS / AMD Radeon RX 5700 XT / AMD Ryzen 7 3700X 8-Core 3.59 GHz / RAM 64 GB
|
|
|
Re: Can it beat t7z though?
[Re: Dullaron]
#359578 - 10/10/16 08:29 AM
|
|
|
There are plenty of compression progs out there that beat 7zip (and t7z with its fixed parameter set). Usually by 5-20% in size. 7zip right now is the 'best balance' for the sort of thing going on in MAME. Speed and size wise. There are some out there that do much better on different types of data. For something like a streaming compression 7zip/deflate is actually a poor choice as it uses too much CPU and memory. There are much better ones out there for that choice. There are also tons of experimental ones out there. 7zip ended up being chosen because it seemed to be popular enough and was opensource enough to include.
Here is a decent recent comparison of the progs/algs http://mattmahoney.net/dc/text.html zstd is the one to look for. It seems to have ~15% worse compression ratio, similar compression times to 7zip, but wildly better decompress times. Its author seems to have designed it for viewtimes in programs. It compresses better than zlib so it is probably looking to be a drop in replacement for that. I would not be surprised at all if there are cases where it beats lzma. But in general size wise lzma seems to win.
Would it be worth adding? Maybe, if you want to optimize to speed of load. But it will cost you in size if you are currently using 7zip and lzma compressed stuff. But you would probably gain some space if you are using zip archives.
Also downside to the original site is he compares to zlib. Which is basically the baseline. It is usually not even as good as pkzip which it emulates. zlib is basically mid 90s 'ok' compression tech. Everyone compares to it because everyone seems to use it.
I personally am not unhappy with the decompression speed/size in MAME. What probably needs to be looked would be something in the rom lookup. From the procmon traces I see it looks to be doing a lot of back and forth opening files/directories and ignoring past results and repeating them. But that is just speculation on my part with a small bit of evidence. Have to dig into the code a bit and see.
Also CHDs is what consumes the most out of everything these days. That would probably be better served if CHDs could inherit from each other like VDI/VHD files do with x86 virtual machines. Sort of like what happened with merged sets a long time ago. Which in itself yielded about a 50% savings. Even then it may not be interesting in the amount saved or work at all in saving anything due to the way these sorts of devices lay data out.
The crux of the issue are there better compressors out there worth using? Lets say you get a modest 5% better compression. That would be well over 100 gig of data. Nothing to sneeze at. But not as interesting when you consider the whole thing stands around 2.5TB right now.
|
|
|
| R. Belmont |
|
Cuckoo for IGAvania
|
|
|
|
|
|
Reged: 09/21/03
|
|
Posts: 9713
|
|
Loc: ECV-197 The Orville
|
|
|
|
Send PM
|
|
|
Re: Can it beat t7z though?
[Re: lharms]
#359597 - 10/11/16 07:25 PM
|
|
|
> Also CHDs is what consumes the most out of everything these days. That would probably
> be better served if CHDs could inherit from each other like VDI/VHD files do with x86
> virtual machines.
CHDs *can* inherit from each other. The Virtual PC VHD file and MAME CHD were invented by the same person, after all (note the similar acronyms). It's just not worthwhile savings for the arcade games we support at the present time.
I do think once we get serious about software lists for e.g. Windows 95 that we'll want a base parent Win95 install and various games and apps will be child CHDs that overlay an install of that item on top of the base OS install.
|
|
|
|
Re: Can it beat t7z though?
[Re: R. Belmont]
#359600 - 10/12/16 01:36 AM
|
|
|
> > Also CHDs is what consumes the most out of everything these days. That would
> probably
> > be better served if CHDs could inherit from each other like VDI/VHD files do with
> x86
> > virtual machines.
>
> CHDs *can* inherit from each other. The Virtual PC VHD file and MAME CHD were
> invented by the same person, after all (note the similar acronyms). It's just not
> worthwhile savings for the arcade games we support at the present time.
>
> I do think once we get serious about software lists for e.g. Windows 95 that we'll
> want a base parent Win95 install and various games and apps will be child CHDs that
> overlay an install of that item on top of the base OS install.
Cool I suspected as much.
I fiddled with it just to see what would happen. The docs were not totally clear how this worked  I used gauntlet legends
chdman -o test.chd -i gauntl12.chd -op gauntleg.chd
~1gig to ~7 meg. Not bad... The cost would probably be semi heavy during mid emulation though.
I then picked a couple of the CD games at random. It was ~2k saved and 2meg as suspected (basically not worth the trouble). If I remember correctly ISOs just linearly drops the data one after another. So if say a file was patched and added/removed enough it would have a different size and move the sectors around after that point. This is the case I was thinking about. But now that I tried and put a bit more thought into it probably would not work very well.
It probably would only work decently on hard drives where there was some sort of update done in place. It would probably need to be from the same machine to have a shot at being kinda close. You might get lucky with two different machines and it smashes up nicely but that would be rare. It might also work if the manufactures had a golden image and updated it in place and did not recreate the whole thing. I suspect the atari/midway games probably would work decently this way.
Yeah the win95 thing will be interesting. As 'just works' is appealing vs 'install windows and then your software'. Both ways have very valid reasons for being done.
|
|
|
Index 
Threaded 
worth a look?: smaller-and-faster-data-compression-with -zstandard
Reply
[Re: 
I used gauntlet legends 