Like many other words, race has two main connotations. One is a scientific term and the other a more common colloquial term. It doesn't really matter if you replace the word with ethnicity as the two, in common speech, are interchangeable. The simple fact, in general terms, is that people tend to congregate along ethnic lines. It's almost a self-imposed segregation. (This is assuming there is not government enforced segregation like in America prior to the 1960's and in South Africa under Apartheid.)
When no government policy is in effect that curtails free movement of the population, economic factors play a large role in where people live. Regardless of race, you live where you can afford to live. As far as white privilege goes, my guess it applies to any ethnic group, put in a position where one is in a distinct minority position, especially so if one is economically challenged. For example, a multimillionaire white person would have a far different experience in Beijing than would a person of modest or low financial standing. Likewise, a wealthy black person in a white majority area, these days, can pretty well do whatever they want with little to no backlash. So, in short, I think the idea of white privilege is over-blown by several orders due to serious mitigating circumstances that muddy the waters.
Yes, the canard being floated is that if you deny white privilege, for any reason, it just demonstrates your lack of empathy for people of colour. Heaven forbid that the idea itself is on shaky ground to begin with.