Good question. Thanks for that.
If worshipping is bowing your head before an idol, then anybody can do it.
However, in Puranic times, Hindus regarded the supreme power as Brahma, the chaitanya, energy, worshipping it meant sacrificing materialism, tapasya, following strict rules (such as no non-veg, intoxicating drinks, bramhacharya etc) which may not be accepted by all. So those who seek true God for self realisation, (Brahma seeker becomes a Brahmin, the caste) can folow that path. Others can learn from his teachings.