Because it's not true.
There are 2 parents that contribute to the genetic make up of the offspring. There's also environmental issues to take into consideration.
If males with brighter colors or stronger odors become appealing to females and those males so described successfully reproduce in a greater proportion to males that don't express those traits or with the same intensity, then those colorful and stinky males have greater fitness.
Those things could be seen as proxies for male vigor, since those males that possess brighter colors or attractive stinks can afford to waste the energy producing them.
This is seen in bird populations. In some species of swallow, those males with longer tails have greater reproductive success and are fitter.
And of course, other factors are involved too. Chance events, zygosity, atavisms, etc. If two parents have a fit phenotype, but are heterozygous for some of the traits that impart greater fitness, they will have a proportion of offspring that don't have those traits and are less fit in the given environment.
Wikipedia had this quote from John Maynard Smith that I find amusing and accurate.
"Fitness is a property, not of an individual, but of a class of individuals—for example homozygous for allele A at a particular locus. Thus the phrase 'expected number of offspring' means the average number, not the number produced by some one individual.
If the first human infant with a gene for levitation were struck by lightning in its pram, this would not prove the new genotype to have low fitness, but only that the particular child was unlucky."
Maynard-Smith, J. (1989) Evolutionary Genetics