The bigger issue is the System, not the people. Open and outright racists are fringe, sure. But the whole governmental, societal and cultural institution has a basis in racism.
The original founders and tenants of American Anthropology, led directly to this, through what we call the creation of Social Darwinism. And permeates our entire System of being. This reliance on Social Darwinism was used to advocate slavery/African inequality and to disposess native Americans of their land. This led directly to Hitler adopting these ideas to promolgate eugenics and genocide amongst "races" he deemed "inferior", and he had "scientific" backing for it.
Social Darwinism is what gives us the ideas that the Poor don't deserve assistance, and that mentally or physically handicapped person's must be miserable and culled.
Edit: "Proponents of Social Darwinism believe(d) that it is/was morally wrong for the government and charity organizations to provide public education, public health, or a minimum wage because these efforts only contribute to the artificial preservation of the weak." - Baker, Lee (Ascension of Anthropology as Social Darwinism)