First off, the distance from "it's said that..." to rumour or old wive's tale is pretty damn short.
Second, if we're engaging in specious generalizations, how about this one: plenty of men are insecure about how they "measure up" in bed. If you think you're a below average lover compared to most men, then your odds of being seen favourably by your partner decrease as the number of men she can compare you against goes up. Women aren't dummies; they can pick up on this.
The worst lover in the world can still be the best man his partner's ever had if she's never been with anyone else. OTOH, he probably knows full well that if his partner's been with a hundred other men, then he ranks 100 out of 100.
OTOH, if a man is confident about his love-making, then the number of men his partner has been with before doesn't matter. He can approach things from the mindset of "you've tried the rest... now try the best."
IMO, this whole notion of "female purity" is just a repackaged version of male insecurity. IOW, it's a polished turd.