Namaste. :namaste
A personal thought and observation? - not necessarily a correct one...
Back in historical times, even the Greeks had their 'pantheon of deities' which included Zeus (the main 'God' associated with thunder and lightning like Indra and/or Thor), Agni, the 'fire deity' was associated with prometheus (fire), Vulcan (metallurgy and alchemy) and hermetic tradition (after the god, Hermes), Poseidon, the water God, and a deity for the sun, moon, certain planets or planetary aspects associated with harvests or festivals - In times of Rig Veda, nature worship (and the personification of 'natural forces') was quite widespread throughout the world, with a similar pantheon of Gods (just with different names, but very similar attributes) popping up all over the world along certain trade routes....faith and beliefs were exchanged and personalised along with goods and reputation.
After a while, the beliefs became more secular and refined away from those 'foreign Gods with different names' to more tribal and local gods and the focus became towards more of a religious and spiritual upheaval revolution (it was probably documented somewhere).
After that, all of the 'gods' lost status and just became 'devas' or 'divine beings' but under the control of a more omnipotent power - I mean, the lightning and thunder may be awesome and terrifying and all that, but that's all it is - lightning and thunder...now what made that happen? Instead of asking 'why'? we should begin to ask 'how?'
Thus, gods like Indra (and later, Brahma) fell out of favour with being worshiped by mainstream Hindus. They became (or were made) 'redundant', 'superfluous' then 'superceded' by 'better models'.
I still include Lord Indra in some of my prayers, in Havan and Nav Graha puja, but I don't pray to him directly. They ever say he is an emanation of Lord Siva anyway.
Om Namah Shivaya