I don't exactly understand calling by inspiration, either, because I've also known women who've been called to work with children when they knew they wouldn't be able to handle a room full of them. To appreciate children is one thing but to teach a number of them takes a certain individual. However, I don't doubt calling by inspiration and believe that people are called for a reason, whatever it may be. Like someone had mentioned before, perhaps the only way a ward could learn something about one of its members is for its leaders to first feel inspired enough to call that person to serve. Just an example.
It's a blessing to be called and a privilege to fulfill it, but each member has his/her life outside of the church and the challenges that accompany it. I don't think the choice to accept or decline a calling can ever be considered wrong or right in the general terms--only wrong or right for that individual at that time of his/her life. Each of us is on our own path, which I believe God understands and is tracking as we go.