Was it "rightfully taken" from the Jews when they lived there? Both sides do have a valid claim to the area. And some sort of right to return does need to be included. But I see a lot of hate and distrust on both sides.
Most Jews in Arabia converted to Islam. Never was taken from them. Medina for example was originally mostly Jews who converted to Islam, then invited Mohammad (s) over and said we will make you governor/leader of us and our city. This how Hijra started.