This is known to happen on Windows XP with MAME and Nvidia cards older than about the 8000 series. I believe a newer OS or newer video card will fix the problem.
The good? news is that there's on-board video on that motherboard, so just removing the nvidia card and using that might work as well. It's original-gen Athlon 64 so MAME performance is going to be, uhh, highly questionable anyway.
Otherwise, you build a PC with junk parts, you take your chances.