This is a very common issue with polarized sunglasses and is specifically the reason I did not get polarized sunglasses last time I was shopping around for some. It was actually hard to find nice sunglasses that were not polarized because many of the good ones only come polarized.