It's probably some hot-key combination like CTRL+SHIFT+some_other_key that's getting hit due to the way your MAME keys are mapped. That combination is triggering some piece of software on your computer. Either something that came pre-loaded from the manufacturer or possibly something you installed off the CD that came with the motherboard.
I know people have had similar issues with the Intel video driver hot-keys that rotate the screen.
Try to pay attention to what exactly you're doing in-game when it happens and see if you can consistently repeat it. Once you know what buttons and/or joystick directions cause it, then look at the default MAME key mapping and figure out what key presses it equates to and try to trigger the software outside of MAME.