All of this has to be considered in a relative context.
The three variables involved, as I see them, are human health, animal well-being, and environmental impact. From a philosophical/altruistic point of view, it's preferable to find a point that optimizes these three variables.
-Certain omnivore arguments don't factor into an ethical debate. "It's natural" isn't an argument for why a reasoning animal should or should not do something. "I like meat" isn't an ethical argument either. Being appalled by improper treatment of a cat or a dog, but to eat factory farmed pork, doesn't make sense.
-Certain vegetarian or vegan positions I've encountered are hypocritical. In my observation, they often don't factor in transportation costs; shipping large amounts of produce around the world probably mitigates the fact that eating lower on the food chain is more resource efficient. Then there are those that don't eat meat, but they eat from farms that are gathered mechanically (resulting in dead small animals), that use pesticides (kill bugs), and that use fertilizers that drain into water and cause environmental damage (like the huge dead zone in the Gulf of Mexico). Plus there are some really non-nutritious vegan or vegetarian diets being promoted.
Although 90-some percent of food animals in the US and probably other developed countries come from factory farms, when it comes to a discussion on ethics, factory farms are basically the worst among the variables I listed. They're not healthy, the treatment of animals is abhorrent, and they're bad for the environment. That's basically the extreme (but common) end of the ethical spectrum of food. There doesn't seem to be much debate there, but that's what almost everyone eats.
Then, apart from that, there is a lot of context to consider. Ancestry can influence nutrition needs (alignment towards meat, inability to digest dairy, etc). Location can be a major factor; the cultures I know of that had vegetarian or vegan movements within them tended to be in warm climates rather than climates with serious winters.
So what it comes down to is, the nature-conscious person who goes out into the woods and legally shoots a deer, or who keeps some chickens, can be far more consistent and ethical than a vegan that transports much of her produce to herself from thousands of miles away and burns all that fuel. And a vegetarian that sticks to local farms for dairy or eggs and produce can be a lot more consistent and ethical than a consumer who eats factory farmed meats and greedily-farmed mass produce. It's rather unattractive for someone to be self-righteous when their position is not nearly optimal. There's almost always some improvement to be made in a given diet for some or all of the previously mentioned three variables.
Some things to consider are:
-Transportation costs (mainly their impact on the environment and consequently the animals in the environment) should never be factored out of the equation.
-Assumptions regarding animal abilities to suffer should be conservative (meaning assume the worst).
-Humans need a vast amount of different substances to thrive, and biological individuality is important to consider.
-All things being equal, eating lower on the food chain is usually a more efficient use of resources.
-The impact of pesticides and fertilizers factors in. A fifth of the continental US consists of cropland, and most of it gets regularly covered in chemicals.
-The method of harvest factors in as well.
-Just because a given diet works for a number of years, or even decades, is not proof that a diet is sustainable over a lifetime or through multiple generations.
-Certain substances, like DHA or EPA (types of omega 3 fatty acid), are primarily found in animal products. They are mainly found in fish. Fish get them from algae, but it's not as concentrated in algae, so rather large amounts of algae would need to be consumed to get recommended levels (but it can still help). Pastured eggs can have DHA as well. Humans can convert ALA (another type of omega 3 fatty acid) into EPA and DHA (and ALA can be found in a few specific plants such as flax seeds), but the conversion process is inefficient and variable between people.
Overall, I think it means a number of diets need to exist on this planet, depending on context (location, economics, individual biology, etc). Vegetarian diets as well as ethical local diets that include animals. Growth of organic crops, a focus on getting as much as possible locally, getting most food from as low as possible on the food chain, raising of well-kept animals for either milk, eggs, or meat, and sustainable fishing and hunting. Most of the activism energy from people that are conscious about the three variables should be directed against factory farming of animals and conventional mass-production of produce.