Jaya Nitaai!
I can give some insight from my school on this topic.
Traditionally varna was assigned by birth because the son of a farmer, would more than likely grow up and learn from his father and become, you guessed it...a farmer. Because such a society was divided into strict classes socially and culturally, your Karma usually determined where you were born (in what family) and hence varna (because your Karma determined your inclinations and qualities in this life). Most of the orthodox schools have very strict rules regarding birth based varna but they still abide by this birth principle. For example, I am considered by birth mleccha (barbarian), so no orthodox school of Vedanta would give me Gayatri Mantra or sacred thread due to my birth. You will find very little support from Dharma Shastra regarding rituals about varna changing (because they simply did not exist). Even the Goswamis (who were fairly liberal) remained fairly silent on this issue so as not to create a big disturbance in society.
It was not until the medieval period of Hinduism that the concept of moving from varna to varna manifested in practice. Srila Bhaktivinoda Thakur (a Gaudiya teacher and reformist) called this system the
daiva varnashrama (Godly varna system) in order to differentiate it from the earlier system. Other schools also started this practice. The system allowed for westerners and also other varnas to accept the sacred and perform deity worship and recite Vedic mantras after accepting Brahmin intitiation from one's Guru (rather than one's father as is tradition). In this system, you had to show that you has specific qualities of that varna, and a superior (such as an advanced Vaishnav or Guru) would himself/herself give you Gayatri Diksha.
I think in our society today, apart from Brahmin diksha, you should not need any special ritual to transition varnas. Varna is pretty much fluid in this regard. As long as you have the qualities for it (i.e bravery and justice as a Kshatriya) you can enter those occupations (lawyer, soldier etc). Only exception would be where a special initiation rite is needed (i.e Brahmin).
Jiva Goswami interestingly gives an additional clause for changing varnas. In order to change varnas from the one at birth (it is possible according to him) you have to chant the Holy Name of Krsna. Because according to scripture, the Holy Name is able to destroy all the blemishes that are associated with birth, and grant one the eligibility for the varna of Brahmin to perform Vedic sacrifices. Bhagavatam reads:
yan-nāmadheya-śravaṇānukīrtanād
yat-prahvaṇād yat-smaraṇād api kvacit
śvādo 'pi sadyaḥ savanāya kalpate
kutaḥ punas te bhagavan nu darśanā
"To say nothing of the spiritual advancement of persons who see the Supreme Person face to face
, even a person born in a family of dog-eaters immediately becomes eligible to perform Vedic sacrifices if he once utters the holy name of the Supreme Personality of Godhead or chants about Him, hears about His pastimes, offers Him obeisances or even remembers Him." (SB 3.33.6)
A person who chants the Holy Names becomes equal to a Brahmin born person (although not necessarily a full fledged Brahman, as that requires sacred thread and Mantras). According to Jiva Goswami, in the earlier society there was no rules or procedure by which someone from another varna could be given Brahmin initiation, even thought those persons were fully qualified to do so as they had chanted the Holy Name. Therefore, those people had to take another birth in order to be given that initiation. But later acharyas introduced a system where Brahmin initiation could be given to anyone provided they were qualified. Sridhara Maharaj explains this nicely:
"We say that by the power of Krsna-Nam, even the impurity attached to one's birth, race, creed, caste, or any other thing; can be done away with completely. It is not possible by yoga, jnana, or any other thing, but by Krsna-Nam, any sort of impurity can be obliterated, even prarabdha-karma. So, when all prarabdha-karma is purified, then one comes to the position of the highest birth, that of the Brahmin. Jiva Goswami says that at this point one comes to the status of a Brahmin boy.
But a Brahmin boy is not considered eligible to do the work of a Brahmin, until and unless he is given the sacred thread and mantram. When he is conferred with upanayana-samskara, then only is he eligible to worship Narayan, perform sacrifice, and carry out other duties which only the Brahmins can do. So by taking the Holy Name one is purified and attains the position of a Brahmin boy. But, Srila Jiva Goswami points out, that because we do not find any system to give the sacred thread to those not born in Brahmin families, then they will have to wait until their next birth.
The Goswami admits that whatever caste one may be, if he takes the Name of Krsna then he discards any defects of his birth, and attains the position of a Brahmin lad. Our Guru Maharaj, continuing this line of thought, said there is no harm in giving them the sacred thread, so the custom may be introduced. Our Guru Maharaj, continuing this line of thought, said there is no harm in giving them the sacred thread, so the custom may be introduced. Our Guru Maharaj came to introduce that."
As for determining varna, I was reading a passage from Chaitanya Bhagavata (biography of Lord Chaitanya) and what they would do is when a child was born or a few months after, they would place few items in front of the child characteristic of each of the 4 varnas, and if the child was inclined by their varna, they would approach that item. For example, items placed in front of Lord Chaitanya was paddy, rice, Bhagavatam, gold and silver, and the Lord gravitated towards the Bhagavatam (scripture) showing He would be a great Brahmin scholar and devotee of the Krsna.
Ultimatley varna is meaningless, if we are not aware of our relationship with God. Sri Chaitanya Mahaprabhu sings:
"
nāhaṁ vipro na ca nara-patir nāpi vaiśyo na śūdro
nāhaṁ varṇī na ca gṛha-patir no vanastho yatir vā
kintu prodyan-nikhila-paramānanda-pūrnāmṛtābdher
gopī-bhartuḥ pada-kamalayor dāsa-dāsānudāsaḥ"
"" 'I am not a brāhmaṇa, I am not a kṣatriya, I am not a vaiśya or a śūdra. Nor am I a brahmacārī, a householder, a vānaprastha or a sannyāsī. I identify Myself only as the servant of the servant of the servant of the lotus feet of Lord Śrī Kṛṣṇa, the maintainer of the gopīs. He is like an ocean of nectar, and He is the cause of universal transcendental bliss. He is always existing with brilliance.' "
You can see below my Guru Maharaj giving Diksha to a devotee couple. You can see the yagna at the end. Nice presentation of the rites surrounding Harinama and Brahmin diksha: