I'm not going to play the hypocrite and pretend the physical appearance doesn't matter. Of course it does. I see people (and everything else) with my eyes first. If I'm going to wake up next to someone, possibly for the rest of my life I want to be able to enjoy the view and he has the right to feel exactly the same way.
I strongly believe that human were created to be attracted to beauty. We like beautiful landscapes, beautiful gadgets, beautiful people, etc.
Do I mean physical appearance is all that matters? Of course not. In facts it gets old quickly. I've met people who were beautiful on the outside and managed to bore me to death very quickly with the extreme emptiness of their heads. I've also met people who, not being particularly attractive at first, became very attractive once I got to know them better due to their personalities.
For me it's all about balance. Mind and body are a package and it's up to each one of us how we want to present ours.