Poor don't pay the Jizyah , they were given money from the Muslims treasury.
Muslims, guided by their religion, continued their civilized giving when they were transformed from Jizya takers to almsgivers to protect and sustain poor dhimmis. Ibn Zangawaih narrated that 'Umar ibn Al-Khattab(R) saw a senile dhimmi man begging, so he said: "We are unfair to you if after this old age we ask you to pay jizya." Then he wrote to his workers prohibiting them from taking jizya from old people.[39] He also commanded: "Reduce the sum of jizya for people who cannot afford for it and give alms to those who are incapable of paying at all."[40]
Caliph ‘Umar ibn Abd ‘Aziz also wrote to his worker in Basra 'Udai Ibn Arta'a saying: "If you find that a dhimmi becomes old, weak and poor, give him [some alms] from the Muslims' Treasury House."[41]
Nevertheless if a dhimmi who can afford to pay jizya refrains from payment, he will be punished without violating his covenant. Al-Qurtubi says:
"It is permissible to punish them if they refrain from payment while such being affordable. However the one proving to be incapable of payment may not be punished because the one who is incapable to pay is exempted and the rich are not bound to pay the jizya for the poor."[42]
http://www.irfi.org/articles/articles_1051_1100/Jizya_in_islam.htm