Note that a 'dot' here means one cluster of red, green, and blue subdots. Using trigonometry the dimensions of your display is around 517 mm x 323 mm. Let's say three pixels of your screen make up one dot. That would give you a total of 640 dots across 517 mm, giving you a dot pitch of 0.8 mm. This is pretty large. Typical dot pitches range from 0.4 to 0.2 mm.
You can't really make your shadow mask any smaller than three pixels per dot because you will get some distortion because you're now blending the individual red, green, and blue texels. So you can either turn the shadow mask off or deal with a more blown up look. Here are the HLSL settings you would use:
This will downsample slot-mask.png by two (the image size is 48x48). slot-mask.png uses six pixels for each dot so you will use three pixels per dot. You won't want to use more than five pixels per dot because that's the point where your shadow mask resolution will be below the game's resolution, assuming 320x240. To figure out this point just divide the effective screen resolution by the game resolution, e.g. 1600 / 320 = 5 and 1200 / 240 = 5.
Let me know if you have any other questions.