It's not a question of society holding people responsible for their thoughts and feelings. It's a question of God holding us accountable for them. And thereby demanding that we hold ourselves accountable for them; whether they end up actually affecting anyone else, or not.
Why would God hold us accountable for something we can't control?
To most theists, wanting to have sex with your neighbor's wife, or daughter, or anyone not your own wife, is considered a sin. And is therefor 'immoral'.
Yes, I know that. The question is
why. If intent is morally relevant, how can I be morally responsible for thoughts and feelings I did not intend?
This is the reason, for example, it's so awful for theists to hold gay people accountable for being gay. It's completely unjust and horrifying, and leads to completely unnecessary and harmful guilt and shame. Yet as you just explained, it follows quite naturally from that kind of theistic thinking. Which is why that kind of theistic thinking is so deeply problematic.
So they seek ways of trying not to engage in the sin of inappropriate desires. To most atheists, this sound silly, and extreme, and they mock theists for it because they judge morality based on external behavior, not on internal desire. And because of this, some theists tend to see the atheist as rejecting the idea of divine morality so they can 'sin freely' in their hearts and minds, even if not in their actions. Few atheists understand this. How could they when they don't accept the reality of a 'divine ideal'. To them it's all just nature and personal choice.
Every atheist who is out of the closet has heard theists tell them they're only atheists to have an excuse to sin. We understand it quite well, thanks.
The other issue is, calling something a "divine ideal" tells us nothing it itself, morally speaking. What is it that makes a thought or action "divinely ideal?" What criteria does God use?
Because they are OUR thoughts and feelings. They are who and what WE ARE.
Actually they are not, in my view. What "we are" is constantly in flux and not under our direct control.
And most theists believe it's their life's work to try and transcend these failings. That's mostly what their religions are all about: helping them transcend their "fallen" (animal) selves, into a more divine reflection-manifestation of their creator-God.
And that is extremely problematic, because identifying a thought as a "failing" will not help you transcend it. It just creates aversion/hatred, guilt, and shame.
You're stuck in the "external" mode of thinking: that this about laws and governments and so on. It's not. Or it's not, usually. Theism is not an external form of government. Its an internal form of governance using "God" as one's divine ideal.
Whether the governance is internal or external, holding people morally culpable for thoughts and feelings is unhelpful and, to be honest, quite cruel. That's generally why atheists reject such things, in my experience.