Fundamentally, a person's opinions can't really be corralled or controlled. They can be "informed," obviously, and change as new information is presented, but even totalitarian regimes cannot truly "control" people's opinions. Maybe their public opinions to an extent.
And regardless, one need only take the idea to an extreme and ask "What if one were the last person on Earth?" Then guess what? Have at it. No matter what opinion you hold, it doesn't matter - you're "entitled" to hold it. So... just because other people happen to be present, that in no way makes the case any different really. You may apply a filter in public, but your opinions are your own, and of course you are entitled to them. No one else can even come close to doing anything "real" about them except try to sway you to their own once they know you are of a different opinion.
You may as well say that people aren't entitled to their own thoughts. That statement is is just as dumb as when "opinions" is substituted. And if Ellison meant "public opinions", then that's what he should have said. There are certainly opinions that will come with their own punishments when cast out into the public sphere - opinions that a person is still "allowed" to hold, and therefore (like thoughts) are entitled to them. In the end I would argue that it's not the opinion itself that is the problem... the problem is when those holding the "ignorant opinions" try to convince others to think the same.