Personally, I see this kind of thinking somewhat dangerous and toxic. Where does it end? Who decides what is what? It's merely opening the door to legitimizing a grievance based victim mentality that should be avoided like the plague.
We shouldn't minimize the suffering of people being actively victimized either. It's really easy to say 'you don't deserve to have these feelings' from on top. The video I mentioned points out a brief deleted scene in Lilo and Stitch which talks about how many Hawaiian natives feel like tourist commodities in their own country after a forced annexation which happened
in living memory.
To them seeing a little middle class white person wearing a sexy hula outfit might cause considerable and not unwarranted cringe, becaus they buy a superficial trinket from a usually white owned company and take it off knowing nothing about the poverty caused by American tourism and sales of their culture. Their interest isn't in the culture, the people, but in a twisted superficial aesthetic.
I'm going to Hawaii over the 4th of July ( not for that reason but because my anniversary is on the 2nd ) and there won't be fireworks or celebration except at the resorts because of a lot of unacknowledged bitterness towards the US by Hawaiian natives. This is a real thing and saying it is just victim mentality really downplays what they're experiencing.
Does that mean all cultural appropriation has negative effects? No, of course not. But, to me, saying cultural appropriation can cause no harm is as dangerous as saying all of it is.