That's how they do it. If you believe in Aliens you're sane. If you believe in Aliens in the bedroom you're insane because that is a rare belief.
No. Rarity has nothing to do with that.
Evidence and rational reasoning has everything to do with that.
Considering everything we know about the universe, physics, chemistry, biology, the nature of life, etc.... It is
extremely plausible, in fact almost a certainty, that extra-terrestial life exists out there, somewhere, among the gazitrillions of earth-like planets that exist in the universe.
Considering the same knowledge, it is
extremely implausible that an alien species stops by earth for a visit.
The justification for both of these, concerns EVIDENCE and DATA. Scientific knowledge.
At no point does it concern "well... many people believe it (or not)".
Again, argumentum ad populum is a
logical fallacy.
When somebody brings to your attention that you invoke a
logical fallacy, the right thing to do is to take it to heart and revise your argument / reasoning to fix the problem. Not to point fingers and say "
well... THEY do it too...." as if that makes it ok for you to use a logical fallacy to make any kind of point concerning whatever.