Yes, because it isn't fair now. The average white person can expect to inherit substantially more wealth than a descendent from slaves. I.e. the inequality is not a thing of the past, it is still here today. White people are still benefiting from slavery.
And it would be exponentially less fair afterwards, although with a different group 'in the lead'. The median white American probably inherits less than $100k.
Even now though, the average Nigerian-American, Japanese-American or (South Asian) Indian America probably stands to inherit more than the average white American. They certainly earn more.
Do they benefit from slavery too?
The average descendent of the Normans in Britain inherits more than the average descendent of the Saxons or Britons.
East Germans I guess inherit a lot less than those from the West. Londoners inherit more than those from Runcorn.
No doubt you'll find a legacy of historical disadvantage in all countries.
$5 million all around?
Black Americans are among the richest demographics in the world with the best opportunities, but they are so disadvantaged that hispanics and Native Americans need to tighten their belts to turn them into the richest demographic in human history?
It's fair to make a hispanic plumber should pay so Jay-Z and Beyonce can get $10 million because of slavery? (you aren't getting $250 trillion to make people multi-millionaires without harming the working class)
You still haven't explained how you would think it would work in practice.