There are a few issues with this:
1) The average fatality rate of COVID is far higher than 0.4% in most countries, and in the countries where it is that low, it is often the case that the majority of the population are vaccinated. Even then, you can't generalize the average fatality rate, which includes younger and healthier people, to everyone who gets COVID given that they may be elderly or have risk factors/pre-existing conditions.
Mortality Analyses - Johns Hopkins Coronavirus Resource Center
2) The average Joe in your example may have multiple times that probability of death if he has pre-existing conditions or risk factors such as being elderly.
3) Death isn't the only problem resulting from COVID. Someone can still get very sick and take up an ICU bed or a ventilator even if they don't end up dying, and if we're talking about health risks as well, the person can also end up having long COVID and dealing with lingering symptoms for months or even a year or more after recovering from the infection.