Yes, of course.... or any gender expression?
Do you think there are many people walking around with swastika tattoos visible who don't also harass people? How big is the sliver of the population you're worried about?
The issue isn't about how many people fall under that category, the issue is assuming and then actively reacting to someone as threatening when there is no clear evidence they're threatening. It would be like an officer assuming a person of color is going to get violent because of their race. Of course somebody can decide not to wear a symbol while nobody can decide their skin color, but my point is that stereotypes are stereotypes. What would stop society from deciding that anyone who wears a trump hat is a racist and should be silenced or responded to with physical violence?
So you agree that, say, a trans or non-binary kid shouldn't get harassed for how they look or dress?
Of course they shouldn't/
Well, that's naïve.
Opinions can become violent, but until they do they're still just opinions. Someone has all the right to prevent someone from coming into their shop if that person is wearing a swastika, but prohibiting them wearing it in public is something I disagree with. Minorities need to be allowed a voice in order to be heard, and it doesn't matter if the minorities are seen as hateful by the majority.