A good way to sum up the imperfect solutions is: "Good men stay out, so bad men stand out" - again, knowing that most violent crimes and sexual assaults are committed by men, in all kinds of places, which is why we have "women's spaces" to begin with (like safe houses, separate women's prisons, separate women's changing rooms, and yes, restrooms). Imperfect yes, but FAR better than the old regime when women had no safe spaces at all, when women were essentially shut out of public life.
Also, "good men" deserve privacy and dignity too. Most "good men" do not want to invade women's personal spaces, and don't particularly want women in their changing rooms, restrooms, etc. either. The men who want to invade women's spaces should raise a huge red flag, and yes, that is why we fight for single sex spaces.