Allowing biological males into women's only safe spaces and women's only sporting events is misogynistic. In the case of safe spaces, of course it contributes to violence.
This claim is entirely contradictory to my experience.
Gender segregation is not an effective way of stopping violence against women - millenia of patriarchal societies have amply proven that. Women have been confined to houses, harems, put away under veils, all in the name of protecting them... and guess what: it didn't work. It was only ever an excuse to limit our access to freedom and power.
My safe spaces have always been inclusive.
The women's cafes, the radical left student house (well, shack... But it was comfy), the queer bars: I didn't feel safe because men had no access. I felt safe because misogyny had no access.
I felt safe, because I knew that in these areas the heteronormative world and the patriarchy were kept at bay. No man would disregard my opinions, or feel entitled to make sexual advances... or if he did, he would be stopped by the others.
Safe spaces are not just physical. They are created by community and shared values.
Any of those "violent men posing as a trans woman" you're talking about (I never met one) would have to cross-dress first (many machos would not even dare to do that) and then insinuate himself into this queer or feminist culture and try to pass as one of us. Unlikely. Sure, anything's possible, but in term of risk assessment? Way low.
And as for sports, there are some non-zero-sum ideas already, other than using one's genitals to determine access to sporting events. Leagues are already looking at hormone levels and maximum of testosterone ( sounds complicated to me, but perhaps for world class events). Or you can simply look at their previous times and achievements and classify people into fair competition groups that way. Pretty sure the Paralympics already do that. Perhaps it would make sports more inclusive, too.