Dairy is one of the things I significantly reduce rather than avoid.
I eat cereal probably 3-4 times per week, and use about an ounce of whole milk each time. I've occasionally used a sprinkling of Parmesan cheese for Italian foods that I've cooked, but often I don't bother.
Generally, I just don't like cheese or milk. My friends tease me because on rare occasions when I eat pizza at some social event, I pull most of the cheese off. To me it's too much, too gross. If it's really fresh and if there isn't a ton of cheese, I'll just eat it as-is, though.
It's also true that dairy is sort of an odd thing. To my knowledge no other adult animals drink milk; especially milk from other species on a regular basis. In earlier times, humans would become lactose intolerant as they grew into adulthood. But, humans domesticated cows in some regions and basically forced their bodies over generations to be more tolerant of milk. But, different ethnicities have different statistics for milk; most Caucasians can comfortably digest milk, but many Africans cannot digest it as well, and Asian statistics are somewhere in the middle. Regions that had domestication, have higher tolerance rates.
Even people that can digest milk, can't digest it as well as other things. Large quantities of milk in short periods of time make almost anybody feel ill, because the body can't digest that lactose and casein as effectively as other things, even for people that aren't noticeably intolerant to small or medium amounts of dairy.
And when I drink milk, I only drink organic milk. Even that I don't trust completely though, for the ethics of how the cows are treated, and how it affects the environment to have that many cows and all of the food, water, housing, and energy needed.
So, I don't really like dairy, there's nothing particular health-related I want from it, it doesn't seem optimal for digestion in any large quantities, I doubt that it's one of the better uses of food resources and ethics, and so I only bother to use it in small amounts for certain things. It would be easy for me to exclude it entirely but I haven't seen a reason to make sure I do that either.