Even the question of 'ownership' of cultural artefacts is highly problematic...
I agree, Augustus. I think this is an issue, along with the perhaps more important question of whether significant harm can be done to someone by "appropriating" artifacts and/or traits from their culture. Although the term, "cultural appropriation" might be meant to be neutral in regard to harm -- that is, to allow for the possibility that any particular act of cultural appropriation can be injurious, benign, or non-injurious -- they very fact that it raises the question is important.
Ultimately, the harm or injury given here seems to be an injury to the ego or psychological self, except in certain cases, such as the physical theft of artifacts. As a matter of injury or harm to the psychological self, I would point to an imperfect analogy. That of speech. It is clear that speech can offend someone's sense of who they are, and it is clear that at least some cases of cultural appropriation can offend someone's sense of who they are. Should we then treat these offenses as more or less the same in terms of seriousness?
As for the notion of ownership, the term "appropriation" might be seen as implying that ownership is possible even though strong arguments could be made against that.
Those are my preliminary thoughts. There's no guarantee they will be my conclusions -- should I ever arrive at some.