My monitor is very sensitive to rolling as well. I had to fiddle with the Vertical Hold a lot until a found a "sweet spot" where most of the resolutions are stable. If you can't find the right spot and it's an older monitor, it's possible that the monitor might need a cap kit or something. As the capacitors get old and dry out, things don't work like they should.
Note that you should only install the Ultimarc tri-sync utility if you have a tri-sync monitor. If you post some clear pictures of the back of the monitor, it's possible that someone here might be able to identify the make/model.
GroovyMAME support forum on BYOAC