You have to set the MAME "pre-scale" to 1 when HLSL is enabled. (it seems to be set to a value higher than 2-3 in the screenshot)
The hlsl color blending and phosphor bleed simulation will work properly then and look more like an arcade-monitor and also less pixelated than in the image. The screenshot also begs for a higher defocus_x & y values
You don't seem to use any "shadow_mask_texture" at all judging from the screenshots or a very very low "shadow_mask_alpha". Only the pincushion effect and a high saturation are visible and it's not doing hlsl any justice.
Cheers
----
“Anyone who has never made a mistake has never tried anything new.” --Albert Einstein