It wouldn't have been against him as a white heterosexual male. But there were pretty sharply divided opinions about him. Nixon could have won the 1960 election if just a few more votes had gone his way; it was very close. America was lot more religious back then, and putting forth a clean, honest, family-oriented image was expected in public life.
There could be any number of possibilities. The House Select Committee on Assassinations suggested a probable conspiracy, but they couldn't really find out much. (
United States House Select Committee on Assassinations - Wikipedia)
It'll probably always be a mystery, as there's too many unanswered questions to be able to say "case closed." It'll just go down as one of history's numerous curiosities and mysteries, and no one may ever know the real truth of it.
Oswald said he was a patsy and effectively denied any guilt. Then he was shot and killed by Jack Ruby who had Mob ties. He was essentially tried posthumously, but never got the opportunity to defend himself.