Given the context of taking diseases on a case by case basis, we should also ask "how frequently is disease X fatal?" - it's part of the calculus.
Generally speaking the calculation when diseases are endemic is a no-brainer: look back at rates of death & serious harm (including lifelong debilities) from measles, mumps, polio etc. and you'd need to be functionally innumerate to think that vaccines had in any way at all made things worse.
There's a much more interesting calculation to be done when a disesase has to all intents been eradicated, because you then have to work out what the possibiility of it being brought back from outside might be: in effect, you stop comparing even the tiny amount of harm from vaccinating against a disease that does a measurable harm, but against the potential harm that it might do, were people not vaccinated. For some, that calculation is still very easy: I think the outbreak in Disney showed how susceptible an unvaccinated population could be to a disease of the infectiousness of measles; for others, especially where global rates are low & travel to and from places where the disease is endemic is minimal, it's a far harder risk to work out.
The biggest problem for vaccination is perception rather than actual harm: as I've pointed out a few times on the other vaccination thread, if you're vaccinating entire populations, the law of averages means there will be adverse events that happen at the same time - adverse events whilch would have happened anyway yet because they occurred slightly after a vaccination, people assume causation (and the reporting of "oh, it wasn't the vaccine after all" is a lot more subdued than the front page "teenager dies after vaccination" story, if it's reported at all)