Sorry. I wanted to edit that post out but the system ate my earlier edit.
It is common knowledge, at this point, that racism not only hinges on easily-discernable phenotypical traits like skin color or hair frizziness, but ascribes a whole host of mental traits and cultural habits onto people based on these physical traits or their "common ancestry" as you put it so nicely.
This ranges from relatively innocuous assumptions such as that people originating from East Asia are good at maths or hard working, to more problematic tropes such as the idea that ethnic Jews are intrinsically talented at handling money, to the rather insidious notion that some races are intrinsically more intelligent or more apt to work at intellectual tasks than others, culminating in pseudoscientific garbage like The Bell Curve, where it is asserted, without evidence, that the reason why Black kids often skew lower on IQ tests has to be genetic and intrinsic to their Blackness, without even bothering to control for other factors.
So when people commonly talk about race, they are often not simply talking about purely physical phenotypical traits; they are carring a host of assumptions and prejudices about mental and cultural traits or habits with them, consciously or not, because that's the connotation the term has acquired over the centuries of its common usage.