Didn't they?
Clement of Alexandria, from the 2nd century, acknowledged that there were men who had a natural aversion to women, and were called to lives of celibacy and shouldn't marry. He said that these men were eunuchs from birth, much like how Jesus said in Matt 19:12.
Matthew 19:12 - For there are eunuchs who have been so from birth, and there are eunuchs who have been made eunuchs by men, and there are eunuchs who have made themselves eunuchs for the sake of the kingdom of heaven. Let the one who is able to receive this receive it.