You may well be right, as far as
averages go, but 50% of any group lies above the mean and 50% below. So it's by no means a sure-fire tell.
But intelligence is not a simple thing, there are many different kinds of intelligence, and many different ways of looking at people. To be honest, I'm not even sure that to say a person is more or less intelligent means much. But even if it does, does that mean analytical intelligence, emotional intelligence...? An American researcher proposed in the '80s that there are nine types of intelligence (infographic below). A doctor/lawyer/etc might well test highly on one or some of these metrics, but a builder, a gardener, a nurse, a cleaner, would likely rank highly on others. IQ is by no means a complete or reliable picture of something as nebulous as intelligence.
And no, I certainly wouldn't advocate discarding anybody's views - be it a doctor's or a bus driver's.