I think this is a very fair point.
What I would say is this: the harm Trump is capable of greatly exceeds the harm he’s actually done to our country. And he’s done a lot of harm. If you can’t see this ... you haven’t been paying attention.
We know this, because Trump has been repeatedly blocked and stymied by the FBI, by the Democrats, his own Party, his own “adults in the room” advisors ... in some cases (like child separation) by his own children. What happens in his next term, now that he’s largely purged his administration of the “adults in the room” and been essentially exonerated for obstruction of justice (Mueller investigation) and for abuse of power (Ukraine)? We are very, very lucky that he is not a war-monger .... but some unforeseen crisis, like COVID-19, was inevitable. And a war in his second term - whether started by us, or not - is still possible. No one can predict what he will do if/when these things happen, but we know we are not well-served entrusting enormous power to a petulant child.
People take far too much comfort in the fact the car hasn’t completely careened off the highway with Trump at the wheel, without appreciating the reason that has happened is because he has repeatedly bumped into guardrails. Those guardrails have been severely damaged in his first term. What happens when they fail? The whole point is to avoid total disaster in the first place, not wait until it happens. Unfortunately in some respects it is already too late; we are seeing our first major failure of Trump with COVID. There will be more.
Remember this is the guy who told stories in 2016 about heroic American servicemen in the early 20th century, who would dip bullets in pigs’ blood before executing Muslim insurgents in the Philippines (no really, he said that - go look it up). He said we need more torture. He said we should target and kill the families of suspected terrorists. Rex Tillerson and other former advisors say they repeatedly had to tell Trump that they could not do what he was asking - it would be illegal - and he would then go into a rage. Trump threatened to bomb cultural sites in Iran - which would be a crime against humanity. In private meetings, he’s asked his advisors why he can’t have more nuclear weapons “like other Presidents had”. He told Woodward that he has a “new super weapon” like no one has seen before. He threatened North Korea, an unstable nuclear power with “fire and fury”. He Tweeted that his Tweets are sufficient notice to Congress and justification for military strikes.
Second, the type of harm Trump is doing is the kind that undermines our democracy going forward. Richard Nixon was going to be impeached on a bipartisan basis - and he still released his tax returns. After he resigned, we still lived in a democracy. LBJ escalated the Vietnam war; the American public debated, protested, voted, and as a result LBJ did not run again. Again, after LBJ, we still lived in a democracy. A democracy is capable of learning, and correcting its errors.
Trump short-circuits this. For example, when he tried to use US military aid to bully an ally into investigating his political rival ... that doesn’t just dent the roof of our democracy. It damages the smoke detectors.
If nothing else, perhaps the lesson we need to learn here is that more Constitutional restrictions should be placed upon the Office of the President, so as to put in better checks and balances. The presidency used to be much more limited, although once the "Imperial Presidency" was established, it elevated the office far higher than it should have been.
I also think that President and Vice-President should be separate elective offices, not on a combined ticket, and I also think the Cabinet posts should be directly elected instead of appointed by the President-Elect. These are things that can be done to help limit the amount of damage a single president can do, and if we don't do that, then we have no room to complain about who gets elected.
As for damage, it depends on what kind of damage and damage to what. In my view, I think the country has been on the wrong track for at least the past 40 years. There has been real, visible harm to this country due to the agenda of Trump's predecessors and their allies - many of whom are Trump's loudest critics today.
I would have preferred an honest and frank dialogue about the policies of this country, our economic situation, our foreign policy, and our national security aspirations. Instead, all we ever really hear about are the horrible things they've done or said, the crooked deals they've been involved in, and various personality defects - on both sides. It even trickles down to ordinary Joes like us discussing politics, where there's this kind of "for us or against us" mentality at work. "If you don't see the world in the exact same way I see it, you must be a communist/fascist/socialist/evil spawn of Satan."
It all seems to be based on emotions and how people feel about a specific, individual politician - and not on the issues of national importance.
I try to look at this methodically. That is, I observe the current situation in America and the world at large, and I think: How did we get to this point? What mistakes were made? Where, exactly, did we go wrong?
Furthermore, the key question is: How do we get ourselves out of this mess? You might say that we should get Trump out of office first, but then what?