Are we all not affected by religious beliefs that are not our own, in some way or another?
How about when religious beliefs get codified into secular laws? If the justification for the law is based on religious belief, cannot those beliefs be evaluated and challenged?
How many conflicts around the world are based or strengthened on religious beliefs? Catholic vs Protestant, Shia vs Suni, Hindu vs Islam, Judaism vs Islam, Christianity vs many others. These conflicts affect everyone.
I understand what you are saying, and I am not unsympathetic. If someone finds peace and personal happiness in their religious belief system and religious practice, whether what they believe is about something real or not, then they should be left to be happy in that belief.
But what to do when it goes beyond the individual; when a religious belief system gets aggregated and amplified by millions of people? How is the non-religious person (or religious person with different views, for that matter) supposed to push back against the institutionalizing of religious beliefs into secular society? How is one to reconcile the opposing desires of multiple religious groups trying to influence the secular in conflicting ways, all being justified by their religious beliefs?
How is one to address and resolve all these challenges without addressing the religious beliefs used to justify conflicting positions?
Most of what you wrote here has to do with personal beliefs affecting others. I made that qualification in my response to you.
Once a religious belief impacts my reality, then it's fair game for me to question or challenge it. If it doesn't, what another believes is not my business (save scholarly debates, as @sayak83 offers in post #29).