Allowing people to hide behind rights they do not accept others having will have horrible consequences down the road. This is the bigger concern. Tolerating the intolerant leads to the end of tolerance for all.
You should forfeit your claim to any rights you advocate taking away from others. There is nothing unethical or morally objectionable about reciprocity, no double standards allowed.
You are confusing "should" with "must". If they willfully give up their rights, that is a lot more reasonable than taking their rights away against their will. You can't rob anyone of their constitutional rights. And, until neo-nazis actually take ACTION rather than mere words and expression, there is nothing that should be done to rob them of their constitutionally protected rights.
Part of living in a free society means putting up with things that are despicable. Neo-nazis are a disgusting breed of human beings (if you can even call them that). But, they remind us of what we must keep fighting against. They remind us that racism, nationalism, and hatred are still a real enemy in the US. The recent election made that very clear as well. It's not just neo-nazis that we must be weary of, it is the alt-right as well. In a lot of ways, they are one and the same. America first is a dangerous policy, and it is one that will harm America in the long run.
But, a free society must have open debate. There cannot be any fear of legal action against one for merely expressing their beliefs. That WILL lead to authoritarianism and inevitably the end of freedom of speech.
In other words, to protect the right of expression for reasonable people, the right of free expression must be protected for the worst of us.