People's rights and freedoms do indeed supercede the irrational and unsubstantiated fears of others.
So if I understand you correctly, the trans people's feelings are legit and should be honored, but women who do not want biological males to share their bathrooms are being irrational. Did I get that correct?