I don't think that man was ever "perfect" or existed as perfect as seen through the eyes of Christianity.
I would agree with this, in the sense of we weren't "superior" at one point in the past and everything became rotten after that, in any actual historical reality. Metaphorically, one could argue otherwise.
What exactly were we supposed to have fallen from?
In the myth, the fall is from Grace, or union with God. So, full Awareness of the Divine, is what became lost and we fell inward into the ego-self, experiencing separation from ourselves and the world, and from ourselves with others, and from ourselves with our own self, a form of death in isolation. That is what is conveyed in the myth as the fall from God.
Is there any evidence of having been better than we currently are?
That condition I just described is self-evident today to anyone living their lives in the world. We are separate from each other and the world and even ourselves with ourselves. It's rampant. In the basic human condition we all share on the planet.
Is there evidence for a different state than all of that? Yes. Absolutely. That is the mystical experience of life itself. It's a fairly common state shared by both believers and non-believers alike. I had one when I was 18 and not part of any religion or religious beliefs. That state is of absolute unity with the world, others, yourself, and everything that exists and beyond.
That experience is the foundation for such myths as you see in Genesis, and why it rings so true once you've had a glimpse at both sides of that condition. It is hell to lose that connection. It is nigh despair if it weren't for the lights that shine through the clouds for us now and then.
We had a barbarous past. We are certainly better today than the past that we know of.
Well yes. That story is actually better understood as a "timeless story", which is the nature of mythology to begin with. Historically of course, we have actually become progressively better! That's the nature of evolution at work. That doesn't just happen to biology, but in areas like human behaviors. We have evolved socially, mentally, and I'd add spiritually as well. There are whole areas of research which show this, such as you'll find reading Jean Gebser.
Maybe Adam and Eve are symbolic of our potential, but I don't believe it is symbolic of our past.
I see them as a human potential in the sense of the divine human. The life divine, is one where the human walks in unity with the divine at all times. There is another word for that, which is Enlightenment. That is a "return" to that state of Oneness, and a return to Source, the Divine which is both the ground and the goal of the evolution of our species. Adam and Eve are archetypes of the Divine Human. And that is what the mystic traditions point out, in so many words.
So you can see, why a literalist reading of the book of Genesis, makes my eyes want to bleed? It's painful, because it taking the myth and making it some ****ing argument for historicity. It's a myth. My God, read it with your eyes opened, for God's sake!
I don't mean you personally, but "you" as in the religious who argue it as history, rather than seeing the higher metaphorical truths that it says, such as I just pointed out above. It makes me want to gouge my eyes out.