For the tow bar (the receiver). Never. 99.9% of vehicles you cannot remove the “receiver”. The receiver is the “tow bar”. Maybe leaving the ball on in some places is a problem. But I have never heard of such a thing. Getting hit without the hitch will do plenty of damage too. Half the vehicles around here have them on. Recreational lake. I think it’s better to hit the hitch on a minor hit. On a major hit it won’t make much difference. That’s the way I look at it.
Can you tell me make and model of whatever hitch it is you have?