You’re right.In argumentation theory, an argumentum ad populum (Latin for "appeal to the people") is a fallacious argument that concludes that a proposition is true because many or most people believe it: "If many believe so, it is so."
This type of argument is known by several names,[1] including appeal to the masses, appeal to belief, appeal to the majority, appeal to democracy, appeal to popularity, argument by consensus, consensus fallacy, authority of the many, bandwagon fallacy, Argumentum ad populum - Wikipedia
The converse of this is that if many or most people do not believe it, it cannot be so, and that is also fallacious.
The reason why arguments like this have bearing, is that few want to be unpopular!