I have something similar to this in a different way. The God that the atheist rejects is the anthropomorphic form of God of traditional theism. To say one is an atheist, is to start and end with that imagining of God as the great being in the sky who looks down up his creation, who acts upon it with either grace and compassion or disciplines and judgments, the way a parent would with a child. In other words, the atheist rejects this mythic-literal interpretation of God, what really boils down to a mode of interpretation of reality. In a sense, God is still a symbol, even to them.
Ahm... well, of course you don't get to speak for all atheists. Not even I, being an atheist myself, get to do that.
That said, I agree with you to a certain degree.
Speaking from my own perspective, well, an anthropomorphic god is way more rejection-worthy than an abstract absolute simply because the notion contains a lot more assertions (and, more often than not, assumptions). The most basic being that there is an absolute person. Who acts, has a will, has attributes. Etc.
But, of course, also because that is what moves people, makes them do stuff, often destructive or at least irrational stuff. And also because the vast majority of discussions we have are with people who believe in just such a god. After all, atheists are emotional beings too.
An abstract absolute with no personal attributes is not very cringeworthy because it's not very impressive. I mean, the concept doesn't really change anything, it has no bearing on our behaviour, our attitudes etc. It's pretty much irrelevant. Insofar, I see no need to reject it with any fervor.
Do I reject an abstract absolute on a purely intellectual level? I don't really know. If you consider Münchhausen's trilema, which I think is a fairly good start (though I'm never sure whether it is actually a trilemma and not just a dilemma), then the chances for an absolute beginning are equal to the chances of an infinite regress. Somehow, positing a beginning seems a little more intellectually unsatisfying than infinite regress, but that's probably just my own predilection and my taste for infinity paradoxes speaking. As I said, a non-anthropomorphic god doesn't make any difference, so it doesn't matter.