It's not so free a market if building restrictions & permit fees prevent affordable housing.
That's a fair point but let us consider the following...
As I've stated before, California suffers from over-population. Can we agree that a city or state has a threshold as far as how many citizens it can efficiently host? Housing is not the only limiting factor. Other infrastructures and qualities have to be considered. Transportation, water, electrical, clean air and so on.
And as I've stated before, city and state officials are artificially limiting housing to establish a base line population. We have to limit our growth and not just let it expand out with ensuring all other infrastructure in place. I've seen cities like Las Vegas and Phoenix expecting a boom only to be left with empty infrastructure and empty housing. This is all social science so it's hard to suggest a good balance but in general, I agree with this slow approach.
So yes, I agree with you but I believe there are valid reasons why we do it.