I think that depends how broadly and in what contexts you're using the term, to be fair. For example, you wouldn't contest that an infant lacks a belief in giraffes, or lacks a belief in the efficacy of a two-state solution in the Israel-Palestine conflict, or lacks a belief that there is a sound solution to Fermat's last theorem. So, in a technical sense, I see no issue with stating that a baby lacks a belief in a God and, therefore, can technically be considered an atheist.
The difference is if one makes the assertion that this meaningfully contributes to discussion rather than simply being an acknowledgement of the broad definition of atheism as being a kind of neutral position that can include lacking belief. In other words, I think it's fine to say "babies are atheists" in a technical discussion about the broadest possible applications of the term, but I don't think it has much utility beyond that simple, observational illustration. When debating theism/atheism, the subject is rarely if ever debated from the position of not even having the ability to assess the claim, so I would say that it almost never serves much utility in any debate that isn't explicitly about the broadest usage of the term "atheist".