A similar problem occurs when determining if strong AI is possible. How do we get from atoms to awareness? We intuitively would reject the idea of a lego turing machine just needing complexity to become aware. Fundamentally awareness is due to cause and effect and our brains are simpy aware of the change in chemisry and other physical elements. That is how atoms are in a sense aware of the environment as the interactions become like footprints that can later be percieved. This would be how simple physics and chemistry can eventually evolve and emerge to complex biology and even sentience. I think memory is a key component of awareness and if the mark doesnt hold it is as if it never happened but doesnt mean the experience doesnt happen. We have all experienced the entire univverse just cant recall but we also gather clues as to wa happened which counts in our overall awareness of our environment. The universe just doing its thing but because of its fundamental properties also changing and learning in the process.
I confess that I have found this a most fascinating concept, and honestly, one we may yet have to confront most realistically even within my own lifetime.
No doubt you have read of an impending "singularity" of sorts, in which humans need decide whether to "pull the plug" upon AI capacities minutes before machines may actually become so vastly "intelligent" as to emulate humans in thinking, or worse (*gasp*), evolve to some higher order of compassion and awareness that surpasses our own...and then what?
The capacity to "think" is really just a mathematical equation of trillions of calculations in any given second of time, and we are now knowledgeable enough to calculate those numbers as very definition of "higher order" thinking.
And the "robots" are catching up, even faster than imagined.
What the canary in the coal mine warns of, re: "AI, " is not a potential mimicry of humanity as described by intellect and self-evaluations..., but instead an utterly unique and new, and terrifyingly incomprehensible "intelligence" that we may not only fail to recognize, but may also have no capacity to subjugate or control.
Within our own species, this "moment" occurs often enough...when we see that moment when our progeny are in fact are "smarter" than we are...
,,,but at least those superior qualities are managed and held in check by the life-long projections of experience, and the resident wisdom garnered by such events, regulated and kept in perspective by measures of success vs failure.
In one sense, perhaps the most valued one, our capacities to learn from our own failures/mistakes over many years of life, lend us our own unique insights in decision making as to how to act in moments of crisis or doubt or uncertainty in confrontation of future events.
But then consider the "AI", the intelligence that is so fast, so "smart", so "wise"...that may exhibit the collective abilities of wisdom/experience not over the course of 70 years, but in only one or two years.
An "artificial intelligence" that has the ability to "calculate" virtually all and any outcomes of any decision allowed to individualistic "choice"?. That quality defines how humans choose "leaders" to best chart a path to continued hope to a more prosperous and cherished "future".
Soon enough, machines will not only match our own human capacities as regards "intellect", but may even match or exceed anyone's exercise of "wisdom"...absent any of those added human attachments of greed, power, doubt, or vanity.
By then of course, it will be too late.