The Right of Khilafah was only for Imam Ali. But others disobeyed Prophet Muhammad. This disobeying is not surprising, they disobeyed Prophet very times. For example,
When Allah's Messenger (s) was on his death-bed and in the house there were some people among whom was `Umar bin Al-Khattab, the Prophet (s) said, "Come, let me write for you a statement after which you will not go astray".
Umar said, " Indeed illness has overcomed the Prophet ( so He does not know what He says! ) and you have the Qur'an; so the Book of Allah is enough for us".* The people in the house differed and disputed. Some of them said, "Come near so that Allah's Messenger (S) may write for you a writing after which you will not go astray," while some of them said what `Umar said. When they made much noise and differed greatly before the Prophet, He( Prophet ) said to them, "Go away and leave me". Ibn `Abbas used to say, "It was a great disaster that their difference and noise prevented Allah's Messenger (S) from writing that writing for them.
-Sahih al-Bukhari , Book 96, Hadith 93
* Holy Qur'an, 53:1-5
By the star when it sets: (1) your companion( Prophet Muhammad ) has neither gone astray, nor amiss. (2) Nor does he speak out of [his own] desire: (3) it is just a revelation that is revealed [to him], (4) taught him by one of great powers, (5)
According to shia belief, Imam Ali never gave the oath of allegiance to them.
But according to your belief, Imam Ali until martyrdom of Hadhrat Fatimah, i.e., 6 months after demise of Prophet, did not give the oath of allegiance to Abubakr.
Umar narrated,
the Ansar disagreed with us and gathered in the shed of Bani Sa`da. `Ali and Zubair and whoever was with them, opposed us.
(Sahih al-Bukhari, Book 86, Hadith 57)
So she[Hadhrat Fatimah, the daughter of Prophet Muhammad ] became angry with Abu Bakr and kept away from him, and did not task to him till she died. She remained alive for six months after the death of the Prophet. When she died, her husband `Ali, buried her at night without informing Abu Bakr and he said the funeral prayer by himself........ `Ali had not given the oath of allegiance during those months (i.e. the period between the Prophet's death and Fatima's death).
(Sahih al-Bukhari, Book 64, Hadith 278)
Their teachings are the teachings of Qur'an and Prophet.
The Messenger of Allah declared in Ghadir Khom:
"It seems the time approached when I shall be called away (by Allah) and I shall answer that call. I am leaving for you two precious things and if you adhere to them both, you will never go astray after me. They are the Book of Allah and my Progeny, that is my Ahlul Bayt. The two shall never separate from each other until they come to me by the Pool (of Paradise)."
(Al-Mustadrak `ala al-Sahihayn, by al-Hakim al-Naysaburi, vol. 3, p. 109.)