Yeah, it's a big no-no to rub a lens without blowing off the debris first. The most common thing is sand and that will scratch even glass, much less plastic. For camera lenses, people typically use a rocket blower to blow off all debris first before cleaning with lens solution and a lens cloth.