Then just who are the enemies? It's you that used the word.
From what I understand from the writings, it is those who fight against Baha'u'llah from a position of authority, they would be an enemy of Baha'u'llah.
Mostly this has been the Divines, those that from an authority of faith fight against the Divinity of Baha'u'llah.
I get that from this passage.
".....Behold the low estate of these men who know full well how I have offered up Mine own Self and My kindred in the path of God and for the preservation of their faith in Him, who are well aware how Mine enemies have compassed Me about, in the days when the hearts of men feared and trembled, the days when they hid themselves from the eyes of the loved ones of God and of His enemies, and were busied in ensuring their own security and peace...."
The enemies in that passage are the Islamic Divines and Rulers that passed sentence on him.
The remaining men of low estate are those that knew the nobility of Baha’u’llah, but then hid themselves for their own security, when the enemies passed judgement, as you note, they also hid themselves from the enemies.
Rejection of Baha’u’llah's Station does not mean one is automatically an enemy.
Regards Tony