I haven't experienced this problem yet, mercifully. There are a few smaller roads near my place that are 2 and 4 lane roads, with just a yellow stripe divider. I've had many cars opposite me and never had the car slow down. However, I have not experienced a large truck opposite me, as most semi's are forbidden on smaller roads near me (signs saying trucks over a specific weight are not allowed).
I do admit that I tense up a little when a larger car/suv/pickup approaches, expecting it might PB, but so far it hasn't. Note: this is on FSD Beta, AP/NoA is only active on freeways around me, which are 8+ lanes and divided.