The money changers were not the rich you think they were. They were con men who were sanctioned by the temple to exchange the various coins into coins that would be accepted for worship and to purchase sanctified animals for sacrifice.
"...Because Jews came from all over the civilized world to worship in Jerusalem, they obviously didn't carry Tyrian coins. This problem was solved by setting up tables in the Temple courtyard with clerks called "moneychangers" (liter ally, "tablers") who would exchange Temple currency for foreign moneys. By the time of Christ this "convenience" apparently grew into a racket for bilking, with large profits being reaped at the expense of visiting worshipers. It was this practice, along with the selling of sacrificial animals for profit, that enraged Jesus to the point of driving everybody out of the holy grounds, charging them with making the Temple " 'a den of robbers'" (Matt. 21:13, R.S.V.).'..."
The Money changers were also allowed into the temple grounds that were supposed to be only for people to come and worship. BTW, every temple in the kingdom was required to send an animal for sacrifice once a year to Jerusalem. Instead of bringing an animal that had to be care for over the sometimes long trek, the outlying towns would purchase an animal that had been blessed by the Sanhedrin priest. Of course, this animal could only be purchased by the coins recognized by the temple, and these coins could be exchange by-yep, you guessed it-money changers at whatever rate they desired. This is what ticked JC off, not the rich-man poor-man BS.
The Money of the Jewish Temple - Ministry Magazine