It also wouldn't be a democracy anymore, if you limit people's right to vote in such a manner. It becomes, no matter how many or how few, a plutocracy, a system based on wealth, not individual citizenship. That may be okay to you, it's not to me.
The US Constitutional system was set up to protect the rights of property holders (about 15 percent of the population at the time), from the majority...it's why we have a representational democracy, rather than direct. Over time, the US has recognized that such a limited-access democracy is in direct contradiction to the founding declared principles of the nation (inalienable rights, including (but not limited to) life, liberty and the pursuit of happiness).
Let's say we go to an "only taxpayers can vote on how their taxes will be spent" system. Aside from the logistical nightmare of determining who gets to vote on which issues, how is that materially any different from the Communist Party's "democracy" in China, where you get to vote so long as you are a member of the Communist Party?