2) Download and unzip the shader files some place. Maybe in a subfolder called "glsl" or "shaders" in your mame/mess folder...
3) Edit your mame.ini or mess.ini to use the shader. Dullaron showed you how. You need to specifically enable glsl shaders with these settings set:
video opengl gl_glsl 1
and set the emulator to use the shader:
And if you're new to mame/mess there are a lot of other settings you might want to set. Like maybe use vsync/syncrefresh/refreshspeed and disable throttle so you get rid of screen tearing. But all of that stuff is normal mame/mess stuff that has nothing to do with glsl shaders.
I don't use a frontend with mame/mess so I can't help you configure any of those.