May seem that way - I assume you have not ridden trackwork? or not to any extent perhaps?
I have never driven a harness horse, but have ridden, both trackwork/trials/schooling - albeit a good while ago now. Also, evented. My then partner, jockey Ron McCann, had driven many harness horses in fast and slow work, and workouts as well, so a fairly balanced assessment I would think, when he was adamant that a harness horse is much, much harder to hold than a ridden horse.
We had quite a lengthy discussion about this, and he explained that, as the driver was so far away from the horse [ as Curious alluded] the only thing that could be done in an out-of-control situation is lean against the horse and do your best!
500 kgs against 80 odd, no brainer who wins. Of course, a quality driver, with consummate skill and judgment, would not put himself in that situation, and good hands and calmness may alleviate the situation anyway.
A rider, on top of his animal, has far more opportunity to coax and cuddle his horse, and encourage it to soften and listen.