As for people leaving a religion, it can often be a great thing. I know people who chose serving the community, like working for Amnestuy International, rather than proselytising. If the 'service' the church offers is proselytizing or fundraising, I can certainly see why people would leave to do something far more useful to humanity like doctors without borders, or many of the other secular charities out there.
But again, every case is unique. In my view it is more often a good thing than a bad thing.
Baha'is believe if religion causes hatred and estrangement between people it is better to be without that religion. In many ways its up to us what we do with our religion. Hopefully it is an opportunity to become a better, more compassionate and giving person.
I agree that its much better to be of service to humanity than to go around aggressively proselytizing being a thorn in everyone's side.