That is my understanding of Khilafat, chosen by Allah (swt). I think people have misunderstood Khilafat to be Caliphate. So my actual question now focuses on passing Imamate directly through bloodline. This is actually also shared in concept of Caliphate where children of the "King" become successors as "head of state".
Imamate is a matter decided by Allah, based on the characterstics of the imam and not based on the bloodline. In this matter immamate and prophethood are the same.
.سورة البقرة - سورة 2 - آية 124
واذ ابتلى ابراهيم ربه بكلمات فاتمهن قال اني جاعلك للناس اماما...
And (remember) when his Lord tried Abraham with (His) commands, and he fulfilled them, He said: Lo! I have appointed thee an Imam for mankind..
If Imamate is decided by bloodline then after the death of Imam Hasan a.s. his son would have became an Imam. but this was not the case. the son of his younger brother became the Imam.
Yes, Allah preferred some families because of their good deeds.
سورة 3 - آية-34- 33
ان الله اصطفى ادم ونوحا وال ابراهيم وال عمران على العالمين
ذُرِّيَّةً بَعْضُهَا مِن بَعْضٍ وَاللَّهُ سَمِيعٌ عَلِيمٌ
Lo! Allah preferred Adam and Noah and the Family of Abraham and the Family of 'Imran above (all His) creatures.
They were descendants one of another. Allah is Hearer, Knower.
However, Seeing that Joseph was the son of Jacob, and Jacob was the son of Issac, who was the son of Abraham, seeing this does not mean that prophethood is based on bloodline. The same is true with the Immamate.