Would Clarence Thomas have been called an Uncle Tom if he was white?
By the above criteria, I don't know how you could claim that any word is inherently racist. (And I think an important point, re grandma, is that even if white people were to dress or behave in the same way, she wouldn't use that word to describe them, because, at base, it is a racially based slur.)
While I obviously don't think that the term "Uncle Tom" is as bad as the N-word, I think the above demonstrates a hole in your argument: that a word must be derogatory for the entire race for it to be considered racist. It also indicates that referencing specific behaviors isn't enough to make it non-racist, and merely, a common insult.
For instance, what if someone only used to the term "wetbacks" to refer to Mexicans who really had illegally crossed the border? It would then be a specific behavior that they were referencing. Does this make the term any less racist? I don't think so.
Sure, but that doesn't mean it also can't be a racial slur.
I think it's also important to consider the things that are being considered disloyal. Sometimes, simply "acting white", is enough to earn the epithet. I think it's inherently racist to shame someone for behaving in a way that doesn't conform to someone's concept of how a person of that race should be performing.