Thana
Lady
Not implying it, stating it outright. Of course sometimes you might be able to tell, but in the majority of cases you would have no idea. You have very like been in close contact with transgender individuals and had no idea. You may have even have used the bathroom with one with out knowing it.
So to ask the question one more time.
Imagine you see someone who to you looks like a man. Picture a man in you mind. Dressed like a man, walking like a man, sounding like a man, smelling like a man. Now imagine that this individual walks into a woman's bathroom. Would that make you feel uncomfortable?
In my experience, one would have to be quite ignorant not to be able to tell the difference. If you can't tell by face (Which about 79% of the time you can) then their shoulders, hips and legs should be able to tell you. And if you still can't tell then their hands and wrists should be enough. Honestly, unless you're born extremely androgynous then your body almost always reflects the sex you were born as.
But obviously you already know my answer for your hypothetical.