Or both not physical offspring from Abraham (myth) but rather bani-Israel was meant metaphorically "who follows me, then he is from me"
And says about Bani-Israel "offspring of those who we carried with Nuh..." and this makes more sense, since, it can't be expected there be only believers from offspring of Jacob (a) waiting for Musa (a).
Believers from all the people who were carried with Nuh (a) makes more sense and that is what is Surah Bani-Israel says.
To me, part of the corruption of the Torah was to make the emphasis on Ahlulbayt of Ibrahim (a) leading to the Ahlulbayt of Haroun (a) to be not about that, but just about physical offspring and that Abraham (a) and his chosen offspring (a) is not about the chosen ones, but regular people who are physically blood.
When it comes to Prophets (a), the closest to them are those who follow them, not those simply born from them. God chooses them an Ahlulbayt they are sent with collectively with but this is a different issue then their actual bloodline being more special then others. God chooses from their bloodline to their be less competition for succession from false leaders claiming leadership or Prophethood, so that, it's easier to cling to the righteous leaders God chooses.
We see when Zakariya (a) fears regarding Isa (a) not being followed and that his inheritors might claim successorship, he prays for Yahya (a) who instead would prepare the way for Isa (a) and so it makes sense that Yahya (a) doesn't have offspring but prepares for his cousin Isa (a).
The Torah was corrupted in this sense, the concept of Ahlulbayt was lost, but Quran emphasized on it and showed that God always appointed an Ahlulbayt in all times.
The wisdom of Abraham and his offspring is not about everyone born from him, but about how God chooses to assign authority and weed out false claimants to false authority.
This is my take, you may disagree.