When I give our Ukrainian guests a lift anywhere, one of them - the mum - is completely unable to remember to use the button rather than the manual release. I've given up reminding her, because it has no effect. (It's not a language thing, she has the best English of the three, and is the only one who is unable to learn). However the window and trim are fine despite her efforts.
When there's power, the manual release does lower the window, but someone who opened the release and pushed the door simultaneously might be able to damage it. I don't think if you just pull the handle but don't put any pressure it's really that big a deal. Maybe I've just been lucky.