Where does society get the right to tell us what we can and cannot do?
This is a rather interesting question that you pose - you're sort of asking this: where does society get the right to give or take rights?
In effect, there seems to be a "might makes right" issue involved, in which case it's society giving itself the right to give and take rights, which is technically what we actually have.
One issue with this question is that it's vague about what is meant by "society", if you want to get more specific about what the source or foundation is for those rights that can be given or taken.
What I mean is the source or foundation could be at the grassroots level of society on one extreme, or some authoritarian group or individual on the other extreme.
How religion plays a role in determining what rights are given or taken is another parameter. For example, in a country like the US, we have the 1st clause of the 1st Amendment in the Bill of Rights, which prohibits merger of church and state (whether or not the government in the US adheres to it is a different question), but places like Iran, Israel, and Saudi Arabia contrast as theocracies or theocratically-based nations.
If the government says that we do not have separation of church and state, then it can send its agents after those who don't adhere to the state religion tp administer some sort of blunt and rather severe penalty; this sort of thing actually happens in Saudi Arabia and other countries controlled by theocratic regimes.
How mighty that might in question is that the government has, as far as I can tell, is a function of the ratio of physical force and arms that the government possesses to what the grassroot individuals possess.
There's that expression that the pen is mightier than the sword & that's true if you're merely referring to influence and persuasiveness. The proverbial pen is really all bark and no bite. When government bans grassroots individuals from keeping and bearing arms, and enforces that ban, it doesn't have to worry about becoming preventing itself from becoming derelict (with its obligations to society), corrupt, or tyrannical, since it now knows that grassroots individuals have been defanged thus rendered harmless to itself.
That's not the result of some deity that's in control, that's just a bunch of oligarchs, crony capitalists, their minions, etc. all making such policy decisions at the helm of government, but in at least some cases they do so by invoking religion.
Now that I think of it, rights don't come from a deity or government, they're only taken away; there are 3 reasons I can think of for rights being taken away:
1. religion
2. socialism/authoritarianism
3. there's a victim of a crime involved
To me, the 3rd one is the only qualified reason for taking away rights, and by "rights", I mean types of actions. For instance, killing an innocent individual (i.e. murder) is a type of action, and government takes away the "right" to that action; that innocent individual is a victim of that action, and laws that ban murder are there to protect victims.
Having to close your shop 5 times a day during prayer times, on the other hand, is not something that's done for the sake of protecting any victim; it's just an arbitrary religious rule.
One could argue that a socialist/authoritarian policy of having a $20/hour minimum wage is there to protect employees from being victims, but one could also argue that it both victimizes employees by pricing them out of existence as well as small businesses that are rendered economically impossible to function (which includes or entails creating or keeping job positions open).
The best way to answer your question is that it's really grassroots individuals as potential victims of crimes coming together and deciding what we can and cannot do is the basis for what gives us the right to be able to do that, but whether or not they have a good basis or reason for the laws they come up with (i.e., religion, socialism/authoritarianism, or there's a victim of a crime involved) is a different question.
Would that not require a higher authority to grant that right?
Nope.
Well, I suppose this question can be responded to with - whichever god that those who say that such rights come from it are referring to, or - pick one (or more), anyone.