I think this is a useful way to look at this topic!
First: There aren't that many trans people, they are a small percentage of the population.
Second: There are a small number of men who have taken advantage of these situations. Male prisoners have declared themselves trans-women, been moved to women's prisons, and assaulted female prisoners. The same has happened in women's shelters. And in women's locker rooms. Lesbian-only gatherings have been disrupted by trans women. Lesbians who do not want to date trans women get "canceled".
Yes, I agree that the number of occurrences has - so far - been small. But so what??? Allowing trans women into women's only spaces negatively impacts ALL women. So how do we do the math? From a utilitarian perspective, it would seem that we should not adversely impact hundreds of millions of women - even a little bit - to accommodate a tiny number of trans women.
If you do some internet searches you'll find that trans people DO get attacked. It's a real problem and I haven't seen anyone on this board deny it. I'm not denying it. But you'll also find women who are adversely affected by allowing men (and trans women), into women's only spaces.
Here's just one article:
What About Transgender Women in Women's Shelters?
Ask yourself this: why did we create women's safe spaces in the first place?