It's surprisingly easy. I had a vibration that I was able to locate to the grill vibrating against the hard plastic of the inside door panel. Not the same problem you are having.
The grill releases from the door panel, but has quite a bit of resistance. I started on the bottom, lying on my garage floor, using a spudger to ease lower right corner out of its bracket. It is tight and makes quite the noise when releasing, but the clips are very robust. I had no breakage. I then worked my way around it releasing the other (I think) three clips. The grill backing (bracket?) is plastic, so it should not deform, but be careful to not tweak it too much when removing. You could end up with a crease in the grill.
I'd be surprised if there were something in front of the speaker, but behind the grill. While I have not removed the entire door panel, it seems to me that it would be hard for something to get in there. It may be behind the door panel itself. Just an uninformed guess...
Everything I have read and my experience has been that all panels in the interior are very removable, but have more resistance than one would think. I have not read of anybody breaking clips.
Good luck!
