Hit an interesting issue. Maybe someone here will have an idea.
I have three displays-- display 1 is using an onboard Radeon HD4200, display 2 is using one of two outputs on my GeForce 9800GT, and display 3 (master display) is using the last output on my GeForce 9800GT hooked to an LCD HDTV.
MAME is set up to use display 2 as display0 at 1600x900 and display 3 as display1 at 1360x768. If I try to use the HLSL config with dual displays turned on, it will show the white border around the screens awhile, then eventually dump to desktop without a word. As I'm using the precompiled build of MAMEUI64, -verbose isn't dumping out anything I can use to track the exact issue. When I cut it down to a single display, the HLSL stuff works exactly as it should.
So, should I assume this thing really wants multimonitor to have the same resolution, or does this just not support multiple displays?