Doms, IMHO, are Chandalas (as the Dom Raja in Varanasi is known as).
Shudras are not doms. They are very much a part of the Hindu society as also a part of 'Purusha's body'.
ब्राह्मणो.अस्य मुखमासीद बाहू राजन्यः कृतः l
ऊरूतदस्य यद् वैश्यः पद्भ्यां शूद्रो अजायत ll
brāhmaṇo.asya mukhamāsīda bāhū rājanyaḥ kṛtaḥ l
ūrūtadasya yad vaiśyaḥ padbhyāṃ śūdro ajāyata ll
The Brahman was his mouth, of both his arms was the Rājanya made.
His thighs became the Vaiśya, from his feet the Śūdra was produced.
Rig Veda: Rig-Veda, Book 10: HYMN XC. Puruṣa.
No Sayak, what one does is important. The four-fold division of the society was based on what they did, irrespective of their financial situation. The Dom Raja in Varanasi may be stinking rich, but that does not make a difference to his being a Chandala (for something real bad that his ancestor must have done). Similarly brahmins, kshatriyas or vaishyas may be rich or poor, that will not make any difference.