I have the same train of thought sometimes. But then again, maybe it really upsets and angers God that much if you do not believe in what God wants you to believe in. Even if we can't comprehend that or don't think it's fair.
That's a possibility, but I don't think God is a human, and I don't think He would deal with rejection in the same way in such a human and, dare I say it, petty way.
I think giving him human wants and desires is anthromorphising Him too much. I feel a lot of people do end up worshipping their own ego, with a god who hates everything they do, and loves everything they do.
I hold this passage from the Bible to be relevant, IMO:
For my thoughts are not your thoughts,
neither are your ways my ways,
declares the Lord.
As the heavens are higher than the earth,
so are my ways higher than your ways
and my thoughts than your thoughts.
-- Isaiah 55:8-9 (NIV Translation)
Considering we are a tiny part of a tiny planet in a tiny galaxy in a massive universe, possibly one of many, I don't think God would care so much about whether or not we choose to worship him or not, much less to punish people in hell for eternity.