There's a scene in the Life of Brian that I feel sums it up rather well. It's the scene where they are listing their complaints and grievances against Rome. And, indeed, the Jews and many, many others had legit grievances. But they also brought roads, it's pointed out, and plumbing, medicine and other things they didn't have before. And that's probably the best way to go about it. Colonialism is bad enough to make some Marxists complain on grounds of the local state being unable to develop a national bourgeois class due to economic repression. But as anyone of then really worse? Seems akin to the victim olympics and likely to be more destructive in the end than helpful. We're some better off after colonization? I doubt it. Merchants, scholars, military and others more likely to be moving around still would have seen new and different things and culture exchanges would still happen. So technology and other things would still spread (perhaps just as we saw ideas moving around when all of society was largely clustered around the Cradle of Civilization).
Roman, English, Ottoman or Mongolian, they had different rules and ways of doing things and could be said to be better or worse in that regard, but ultimately those conquered by an empire are likely to be looked upon as slaves, exploitable resources, something to abuse and less than a fully entitled human.
Yeah, we can't change the past regardless of how good or bad we consider it to be. In my opinion, pondering what the world would be like if colonialism (European or otherwise) had never existed is similar to pondering what life on Earth would be like if dinosaurs had never gone exinct, or if Pangaea had never broken apart. We will never know for sure, and we can only work with the present either way.