Some people will argue that people are born inherently atheist
People will argue anything and everything. The question is first whether or not the question is meaningful to ask, and second if so what is the best way to approach an answer. Rather than reiterate again the problems with labeling infants atheists, we can turn to the question in a form that is actually studied: are people pre-disposed to finding god or do they require a nudge or even shove in that direction?
It has long been argued, although not without criticism, that in fact theism (or at least religious belief) is natural and atheism is not. For example:
Boyer, P. (2008). Being human: Religion: bound to believe?.
Nature,
455(7216), 1038-1039.
(Unfortunately, I couldn't find a link to a free copy, but anybody who wishes one may feel free to PM and I will make it available through email)
Bloom, P. (2007).
Religion is natural.
Developmental science,
10(1), 147-151.
Geertz, A. W., & Markússon, G. I. (2010).
Religion is natural, atheism is not: On why everybody is both right and wrong.
Religion,
40(3), 152-165.
A key issue in the cognitive science of religion, evolutionary psychology, and to a lesser extent fields like neuroscience is why there is at least a good deal of evidence for the predilection towards religious orientation. Although "just so" stories abound, these are not easily tested. One problem is that the natural controls for a study of cognitive differences between the religious and non-religious is belief.
Studies designed to determine whether atheists relied on different neural mechanisms and cognitive processes than do theists for relevant tasks, judgment, etc.., generally indicate that atheism exists (from a cognitive & neuroscience perspective) on the same continuum of belief that theism does. In fact, some have gone so far as to suggest that, whatever self-reports indicate, atheism doesn't actually exist. There are not many serious researcher (as far as I know) who believe this ridiculous hypothesis. However, that from a cognitive-neurobiological perspective atheism is largely indistinguishable from theism is far better documented. For example:
Johnson, D. (2012).
What are atheists for? Hypotheses on the functions of non-belief in the evolution of religion.
Religion, Brain & Behavior,
2(1), 48-70.
Norenzayan, A., & Gervais, W. M. (2012).
The origins of religious disbelief.
Trends in cognitive sciences.
While it is (IMO) more than a little ridiculous to claim atheists don't actually exist, there is evidence self-reports are not always reliable. For example, one study found that despite differences between atheists and believers in their the verbal responses to statements about god, biological markers of stress/anxiety/discomfort was the for atheists as it for the religious:
Lindeman, M., Heywood, B., Riekki, T., & Makkonen, T. (2013). Atheists become emotionally aroused when daring God to do terrible things.
International Journal for the Psychology of Religion.
Also, when atheist are presented with priming material (vignettes, images, and similar stimuli) of a religious nature they tend to react as we would expect believers do when confronted with opposing religious views:
"In the author’s data, nonbelievers implicitly presented with religious primes showed increases in error-related brain responses. Believers implicitly presented with alternative ideology worldview primes (non-supernatural or otherwise) should show increases in error-related brain responses, a condition not tested by the authors."
Haque, O. S., Shenhav, A., & Rand, D. (2011). Differences in cognitive style, emotional processing, and ideology as crucial variables in understanding meaning making.
Religion, Brain & Behavior,
1(3), 223-225.
Also, for the religiously inclined positive stimuli can aid task performance in ways that do not hold true for atheists:
Inzlicht, M., & Tullett, A. M. (2010).
Reflecting on God Religious Primes Can Reduce Neurophysiological Response to Errors.
Psychological Science,
21(8), 1184-1190.
And of course there are those factors that surprise no one, such as the fear of death. Evoking thoughts of death not only tends to strengthen religious convictions but also denial of the validity of alternative belief systems:
Vail, K. E., Arndt, J., & Abdollahi, A. (2012). Exploring the existential function of religion and supernatural agent beliefs among Christians, Muslims, Atheists, and Agnostics.
Personality and Social Psychology Bulletin,
38(10), 1288-1300.
(once again, my apologies for being unable to link to a free version but as before I'm more than happy to email a copy to anyone who wishes)
Similar positive effects, such as reduction in distress or anxiety, are also possible reasons for a tendency towards religiousity. In fact, it could be that these effects are not just effects of religion but related to a neurobiological predisposition to religion, as some evidence suggests greater activity in emotional control networks among theists compared to non-believers:
Inzlicht, M., McGregor, I., Hirsh, J. B., & Nash, K. (2009). Neural markers of religious conviction.
Psychological Science,
20(3), 385-392.
especially as both theists and non-believers, even when giving completely opposite responses during tasks, tend to use (again) the same regions where activity appears to differ is again in emotional regulation networks:
Harris, S., Sheth, S. A., & Cohen, M. S. (2008). Functional neuroimaging of belief, disbelief, and uncertainty.
Annals of neurology,
63(2), 141-147.
Harris, S., Kaplan, J. T., Curiel, A., Bookheimer, S. Y., Iacoboni, M., & Cohen, M. S. (2009).
The neural correlates of religious and nonreligious belief.
PLoS One,
4(10), e7272.
Other studies indicate other unsurprising results, such as a tendency towards greater reliance on intuitive means vs. analytical in religious vs. non-believers. However, the difficulties in developing a reliable experimental protocol to ensure the results are actually at least related to the variables measured are nothing compared to trying to speculate whether or not such correlations can inform evolutionary psychology. Moreover, in addition to the usual difficulties that confront the cognitive sciences studies designed to explore differences among non-believers face the same challenge as in this thread: it is much more difficult to define someone as belonging to a religion than it is to determine e.g.,. whether a non-believer is a "negative atheist" (or one of a number of similar terms) vs. a skeptical/positive/strong atheist. And, as noted above, self-reports are not always reliable (which we've known about for decades when it comes to the religious, but has not been much investigated among non-believers)
The main point, though, is that a convergence of several lines of evidence seems to suggest a predilection not towards non-belief but belief.