When slavery was first introduced in the colonies, it wasn't race-based nor was it a life-long or inherited situation. It was more like indentured servitude and it was for a limited period of time. It transformed into what it later became in the latter half of the 17th century, possibly triggered by Bacon's Rebellion. Poor farmers, both black and white, had a shared grudge against the elite back then, and ended up fighting side by side against the wealthy. Although the rebellion was put down, the outcome eventually led to slavery becoming race-based and far worse than when it actually began. The elite had a shared interest in not wanting poor blacks and poor whites to cooperate or band together, so racism was invented for that very reason.
This doesn't get anyone off the hook, at least as far as "blame" goes. However, I think it's erroneous to blame "whites" as an entire group, since that pretty much feeds into the same idea which keeps people apart and different groups divided. If any group should be blamed, it should be capitalists, regardless of their race, creed, color, gender, or sexual orientation.