I think that if we're being fair to materialism and non-materialism, the division breaks down like this:
- materialists incorporate facts and ideas into their worldview based on what they're convinced of, and they interpret these facts to all make up one general (and perhaps messy and often counter-intuitive) whole, "the material" (or "the natural").
- non-materialists also incorporate facts and ideas into their worldview based on what they're convinced of, and they interpret these facts to make up two distinct parts of a dichotomy: "the material" and "the immaterial" (or "the natural" and "the supernatural"). Within each of these categories, things may be messy and often counter-intuitive, but there's still a clear divide between what is material and what is immaterial, and both actually exist.
So maybe it's just an aesthetic preference. At the very least, it seems to me that any claim that materialism isn't viable would have to be supported by demonstrating a real divide between "the material" and "the immaterial" in facts that are generally accepted.
Materialism definitely isn't a matter of factual bias, because any "supernatural" claim could be incorporated into a materialist worldview, whether it's ghosts, psychic ability, or the idea that our consciousness resides somewhere other than our brains. With any of these, we're talking about - if the claims are true - real effects that have manifestations in physical effects, so they could be accepted by materialist. The materialists would just presume that the effects are expressions of physical phenomena, and once the phenomena are better understood, the materialist worldview would just expand to accommodate them.
If we're being fair to both sides, then the disagreement between materialists and immaterialists should be like disagreements between biologists about whether two parts of a population form distinct subspecies, or the two "subspecies" are better understood as just examples of variation within a species that should be considered all together.
It isn't really about evidence or which claims a person is going to accept. I think too often, "materialism" is just a label that people with poor standards of evidence put on people who don't accept their claims. It's sour grapes: "he thinks my beliefs are faulty, so there must be something wrong with him."
If two people have the same standards of evidence but one is a materialist and one isn't, they'll both accept the same claims as factual; the only difference is whether they'll categorize those factual claims into one big group or two separate groups. That's it. When a materialist rejects a claim that a non-materialist accepts, it isn't about anything inherent in materialism; it's because the two people have different standards of evidence (or are privy to different information).