I have had that happen in the past with a 120mm fan on the side panel of the case. Normally what your experiencing happens with larger fans like your case fans, CPU fan, or PSU fan. Since most fans on gfx cards are tiny it's really unlikely it could be that, but not impossible.
You can easily rule out the fan on the GFX card by adjusting the fan speed in software (EVGA Precision)
Perhaps look at hardware screen in BIOS to see which fans are active and temps. It would be good if your CPU fan has a fan controller so you can adjust the speeds and see the results of that.