I think a good portion of transphobia is caused by cisgender men not wanting to be gay. I think that's also why there is such a fixation on binary transgender women rather than transgender men. I think internalized homophobia and biphobia has more to do with transphobia than people might think.
I do think there is also some scare-mongering among conservative women about transgender women being sexually deviant men in disguise, which is rather ignorant, and there is a very small portion of misandrists and female chauvinists with loud voices that we know as "TERFs."
And I think there is quite a bit of hostility from gay and bisexual people because they are afraid of being brought down with transgender people, not unlike how Irish people sought acceptance into white spaces in the US by distancing themselves from black people.
There is some ignorance about the fact that gender exists and that some people really are transgender, as well as prejudicial stereotypes and myths about transgender people such as the idea that transgender women are actually just sexually deviant men with autogynephilia, but I think this misinformation is only spreading because people are already hateful for one of the above reasons.
Non-hateful people have been convinced to agree with transphobic narratives because they genuinely do not know any better or, just as often, are trying to conform to the social groups they are in which have stronger opinions on the topic than they do. Once you get them on board with hate, it's as simple as reinforcing simplistic us vs them narratives. Any question of transgender rights, even something as simple as letting them use the restroom, instead becomes a partisan issue about "protecting" women from "those evil men in dresses."
At the core, though, the problem is that people with significant voices do not actually want inclusivity. They are genuinely hateful and tribalistic. The trick is to reveal that to everyone they've bewitched under their spell of mindless paranoia and outrage.