I don't agree, but as you say. The concept of gender would be pretty pointless (and probably wouldn't have been developed in human cultures at all) without sexes to attach it to.
I understand your perspective, but consider the variety of gender identities recognized across different cultures and societies, and the fact that many people's gender identities don't align with their biological sex. Gender is complex and multifaceted, and while it often intersects with biological sex and reproduction roles, it's not strictly determined by these factors. It's a deeply personal aspect of identity that encompasses a range of experiences, roles, and expressions. The development and understanding of gender across human cultures have been influenced by a variety of social, psychological, and cultural factors, not just biological ones.
Let's take a hypothetical where we were not primates, but a social species more closely related to whiptail lizards - one of the few parthenogenic vertabrates. We would all be female and capable of reproducing, but any indivisual could only reproduce by being topped in a pseudo sex act by another individual. Without only one sex, gender roles around nuturing, agriculture, trade, conflict, etc would all still be niches to be filled.