I think in part it is based out of ignorance. This is the same reason why people hate science, any other religion, or really any idea that is not theirs (I am generalizing here).
Some people have a tendency to look at the evil that has been committed in the name of an idea or by those people of an idea. We see it in this thread as well, by people naming off things such as the Crusades. The problem is that people don't sit down and try to figure out what the real reason for these issues were. And then they ignore the good that has been done in the name of the religion.
By focusing on just the extremes really is not a logical thing to do. We could do that with science as well. We could look at the studies of eugenics and see what came of that. However, it is distasteful and simply asinine.
Basically, it boils down to a lack of toleration.
*As a note, I am in no way saying that we should ignore the evils that have been committed in this world. That would be senseless. However, I propose to actually study those evils, see the various factors that played a role, and then try to actually understand them instead of making baseless assumptions motivated by hate.