There are much tougher laws on this in the UK and if you walked around like those guys you wouldn't last 5 minutes. Nobody would be afraid, they'd just call the police and have you arrested or more likely have their mates beat you up. This kind of thing is incredibly serious. We don't have the free speech laws that would allow it.
The part about others' reactions brings another question to mind: in the UK in general, what do you think most people's reactions would be if they saw someone walking around wearing a T-shirt with communist symbols? I realize there may be regional variations in reactions to such things, so feel free to narrow down the question to one country or city for the sake of accuracy if you prefer.
(On a related note, I agree with treating Nazi expression as hate speech like much of Europe does.)