First you have to define what is a close family member? It can make a pretty big difference depending on what culture you are in, especially when trying to define incest.
Other than moral perspectives there is nothing wrong with it. And while the birth defects thing is true, it's less than a half truth. If a brother and sister have a child together, assuming they are both healthy, have about the same the chance of having a child born with birth defects than any other randomly selected child. It's after multiple generations that inter breeding MIGHT cause a genetic problems. There is an equal chance though that beneficial genetic mutations will be passed down. There may also be nothing good or bad. Eventually a homogeneous breeding group can produce genetic clones that lack any genetic flaws.