If you plan on using an arcade monitor, avoid Windows 7.
Windows 7 has some issues with resolution switching from interlaced to non-interlaced and vice versa. There's also a problem with the refresh rates for interlaced resolutions that causes vector games to run at 50% speed. I think this is related to using sync refresh and Windows reporting the wrong refresh rates.
I don't know for sure that Vista has the same issues, but the odds are very likely that it does.
I believe that the current best choice OS for a dedicated MAME cabinet using an arcade monitor is Windows XP x64.
If you're not using an arcade monitor, then Windows 7 x64 is probably the best choice.
GroovyMAME support forum on BYOAC